Abstract:Fine-tuning pre-trained models has been ubiquitously proven to be effective in a wide range of NLP tasks. However, fine-tuning the whole model is parameter inefficient as it always yields an entirely new model for each task. Currently, many research works propose to only fine-tune a small portion of the parameters while keeping most of the parameters shared across different tasks. These methods achieve surprisingly good performance and are shown to be more stable than their corresponding fully fine-tuned counterparts. However, such kind of methods is still not well understood. Some natural questions arise: How does the parameter sparsity lead to promising performance? Why is the model more stable than the fully fine-tuned models? How to choose the tunable parameters? In this paper, we first categorize the existing methods into random approaches, rule-based approaches, and projection-based approaches based on how they choose which parameters to tune. Then, we show that all of the methods are actually sparse fine-tuned models and conduct a novel theoretical analysis of them. We indicate that the sparsity is actually imposing a regularization on the original model by controlling the upper bound of the stability. Such stability leads to better generalization capability which has been empirically observed in a lot of recent research works. Despite the effectiveness of sparsity grounded by our theory, it still remains an open problem of how to choose the tunable parameters. To better choose the tunable parameters, we propose a novel Second-order Approximation Method (SAM) which approximates the original problem with an analytically solvable optimization function. The tunable parameters are determined by directly optimizing the approximation function. The experimental results show that our proposed SAM model outperforms many strong baseline models and it also verifies our theoretical analysis.

Parameter-efficient fine-tuning of large-scale pre-trained language models

Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Models

ADT: an Additive Delta-Tuning Approach for Parameter-Efficient Tuning in Pre-Trained Language Models

Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment

Small Pre-trained Language Models Can Be Fine-tuned As Large Models Via Over-Parameterization.

Arbitrary Few Parameters Are Good Enough for Adapting Large-scale Pre-trained Language Models

Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning

Making Parameter-efficient Tuning More Efficient: A Unified Framework for Classification Tasks.

Parameter-efficient Tuning for Large Language Model Without Calculating Its Gradients

OpenDelta: A Plug-and-play Library for Parameter-efficient Adaptation of Pre-trained Models

An Empirical Study on Parameter-Efficient Fine-Tuning for MultiModal Large Language Models

Fine-tuning Happens in Tiny Subspaces: Exploring Intrinsic Task-specific Subspaces of Pre-trained Language Models

Fine-Tuning Pre-Trained Language Models Effectively by Optimizing Subnetworks Adaptively

When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methods

On the Effectiveness of Parameter-Efficient Fine-Tuning

Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey

Non-Intrusive Adaptation: Input-Centric Parameter-efficient Fine-Tuning for Versatile Multimodal Modeling

Learning Global Controller in Latent Space for Parameter-Efficient Fine-Tuning

Towards Better Parameter-Efficient Fine-Tuning for Large Language Models: A Position Paper

Towards a Unified View of Parameter-Efficient Transfer Learning

Parameter Efficient Fine Tuning: A Comprehensive Analysis Across Applications