Abstract:Large models represent a groundbreaking advancement in multiple application fields, enabling remarkable achievements across various tasks. However, their unprecedented scale comes with significant computational costs. These models, often consisting of billions of parameters, require vast amounts of computational resources for execution. Especially, the expansive scale and computational demands pose considerable challenges when customizing them for particular downstream tasks, particularly over the hardware platforms constrained by computational capabilities. Parameter Efficient Fine-Tuning (PEFT) provides a practical solution by efficiently adapt the large models over the various downstream tasks. In particular, PEFT refers to the process of adjusting the parameters of a pre-trained large models to adapt it to a specific task while minimizing the number of additional parameters introduced or computational resources required. This approach is particularly important when dealing with large language models with high parameter counts, as fine-tuning these models from scratch can be computationally expensive and resource-intensive, posing considerable challenges in the supporting system platform design. In this survey, we present comprehensive studies of various PEFT algorithms, examining their performance and computational overhead. Moreover, we provide an overview of applications developed using different PEFT algorithms and discuss common techniques employed to mitigate computation costs for PEFT. In addition to the algorithmic perspective, we overview various real-world system designs to investigate the implementation costs associated with different PEFT algorithms. This survey serves as an indispensable resource for researchers aiming to understand both the PEFT algorithm and its system implementation, offering detailed insights into recent advancements and practical applications.

FedPEAT: Convergence of Federated Learning, Parameter-Efficient Fine Tuning, and Emulator Assisted Tuning for Artificial Intelligence Foundation Models with Mobile Edge Computing

Exploring Parameter-Efficient Fine-Tuning to Enable Foundation Models in Federated Learning

MEFT: Memory-Efficient Fine-Tuning through Sparse Adapter

FedAWS: A Federated Tuning Approach with Adaptive Weight Shrinking for Pre-trained Foundation Models

Beyond Fine-Tuning: Efficient and Effective Fed-Tuning for Mobile/Web Users

See Further for Parameter Efficient Fine-tuning by Standing on the Shoulders of Decomposition

FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learning

FedPT: Federated Proxy-Tuning of Large Language Models on Resource-Constrained Edge Devices

Efficient Federated Prompt Tuning for Black-box Large Pre-trained Models

Parameter Efficient Fine Tuning: A Comprehensive Analysis Across Applications

FedPFT: Federated Proxy Fine-Tuning of Foundation Models

Rethinking Efficient Tuning Methods from a Unified Perspective

Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey

Federated Learning of Large Language Models with Parameter-Efficient Prompt Tuning and Adaptive Optimization

When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methods

Communication-Efficient and Tensorized Federated Fine-Tuning of Large Language Models

Discovering Long-Term Effects on Parameter Efficient Fine-tuning

PETapter: Leveraging PET-style classification heads for modular few-shot parameter-efficient fine-tuning

Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruning

Delving into Parameter-Efficient Fine-Tuning in Code Change Learning: an Empirical Study

Pluto and Charon: A Time and Memory Efficient Collaborative Edge AI Framework for Personal LLMs Fine-Tuning