Abstract:Large models represent a groundbreaking advancement in multiple application fields, enabling remarkable achievements across various tasks. However, their unprecedented scale comes with significant computational costs. These models, often consisting of billions of parameters, require vast amounts of computational resources for execution. Especially, the expansive scale and computational demands pose considerable challenges when customizing them for particular downstream tasks, particularly over the hardware platforms constrained by computational capabilities. Parameter Efficient Fine-Tuning (PEFT) provides a practical solution by efficiently adapt the large models over the various downstream tasks. In particular, PEFT refers to the process of adjusting the parameters of a pre-trained large models to adapt it to a specific task while minimizing the number of additional parameters introduced or computational resources required. This approach is particularly important when dealing with large language models with high parameter counts, as fine-tuning these models from scratch can be computationally expensive and resource-intensive, posing considerable challenges in the supporting system platform design. In this survey, we present comprehensive studies of various PEFT algorithms, examining their performance and computational overhead. Moreover, we provide an overview of applications developed using different PEFT algorithms and discuss common techniques employed to mitigate computation costs for PEFT. In addition to the algorithmic perspective, we overview various real-world system designs to investigate the implementation costs associated with different PEFT algorithms. This survey serves as an indispensable resource for researchers aiming to understand both the PEFT algorithm and its system implementation, offering detailed insights into recent advancements and practical applications.

HUT: A More Computation Efficient Fine-Tuning Method With Hadamard Updated Transformation

Advancing Parameter Efficiency in Fine-tuning via Representation Editing

Parameter Efficient Fine Tuning: A Comprehensive Analysis Across Applications

Parameter-Efficient Fine-Tuning via Selective Discrete Cosine Transform

Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey

Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning

Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment

LoRETTA: Low-Rank Economic Tensor-Train Adaptation for Ultra-Low-Parameter Fine-Tuning of Large Language Models

GPT vs RETRO: Exploring the Intersection of Retrieval and Parameter-Efficient Fine-Tuning

From PEFT to DEFT: Parameter Efficient Finetuning for Reducing Activation Density in Transformers

See Further for Parameter Efficient Fine-tuning by Standing on the Shoulders of Decomposition

AutoPEFT: Automatic Configuration Search for Parameter-Efficient Fine-Tuning

Unlocking Parameter-Efficient Fine-Tuning for Low-Resource Language Translation

Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologies

UPetu: A Unified Parameter-Efficient Fine-Tuning Framework for Remote Sensing Foundation Model

PARA: Parameter-Efficient Fine-tuning with Prompt-Aware Representation Adjustment

SPAFIT: Stratified Progressive Adaptation Fine-tuning for Pre-trained Large Language Models

Parameter-Efficient Fine-Tuning Method for Task-Oriented Dialogue Systems

PETapter: Leveraging PET-style classification heads for modular few-shot parameter-efficient fine-tuning