Abstract:Large Language Models (LLMs) demonstrate impressive capabilities to generate accurate code snippets given natural language intents in zero-shot, i.e., without the need for specific fine-tuning. While prior studies have highlighted the advantages of fine-tuning LLMs, this process incurs high computational costs, making it impractical in resource-scarce environments, particularly for models with billions of parameters. To address these challenges, previous research explored In-Context Learning (ICL) as a strategy to guide the LLM generative process with task-specific prompt examples. However, ICL introduces inconveniences, such as the need for designing contextually relevant prompts and the absence of learning task-specific parameters, thereby limiting downstream task performance. In this context, we foresee Parameter-Efficient Fine-Tuning (PEFT) techniques as a promising approach to efficiently specialize LLMs to task-specific data while maintaining reasonable resource consumption. In this paper, we deliver a comprehensive study of PEFT techniques for LLMs under the automated code generation scenario. Our comprehensive investigation of PEFT techniques for LLMs reveals their superiority and potential over ICL across a diverse set of LLMs. Additionally, we demonstrate the extended capabilities of PEFT, showcasing its ability to learn from two distinct datasets jointly without compromising performance. Furthermore, our study highlights the potential for tuning larger LLMs and significant reductions in memory usage by combining PEFT with quantization. Therefore, this study opens opportunities for broader applications of PEFT in software engineering scenarios. Our code is available at <a class="link-external link-https" href="https://github.com/martin-wey/peft-llm-code/" rel="external noopener nofollow">this https URL</a>.

Interweaving Memories of a Siamese Large Language Model

Enhancing Large Language Model with Self-Controlled Memory Framework

Unleashing Infinite-Length Input Capacity for Large-scale Language Models with Self-Controlled Memory System.

MemLLM: Finetuning LLMs to Use An Explicit Read-Write Memory

MEMORYLLM: Towards Self-Updatable Large Language Models

Augmenting Language Models with Long-Term Memory

SLIM: Let LLM Learn More and Forget Less with Soft LoRA and Identity Mixture

An Empirical Study on Parameter-Efficient Fine-Tuning for MultiModal Large Language Models

Non-Intrusive Adaptation: Input-Centric Parameter-efficient Fine-Tuning for Versatile Multimodal Modeling

Layer-wise Importance Matters: Less Memory for Better Performance in Parameter-efficient Fine-tuning of Large Language Models

Navigating the Dual Facets: A Comprehensive Evaluation of Sequential Memory Editing in Large Language Models

Training Language Models with Memory Augmentation

A Parameter-efficient Language Extension Framework for Multilingual ASR

MEFT: Memory-Efficient Fine-Tuning through Sparse Adapter

Personalized LLM Response Generation with Parameterized Memory Injection

Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsal

Make Pre-trained Model Reversible: From Parameter to Memory Efficient Fine-Tuning

Learn or Recall? Revisiting Incremental Learning with Pre-trained Language Models

Exploring Parameter-Efficient Fine-Tuning Techniques for Code Generation with Large Language Models

Selective State Space Memory for Large Vision-Language Models