Abstract:Fine-tuning large pre-trained language models on downstream tasks has become an important paradigm in NLP. However, common practice fine-tunes all of the parameters in a pre-trained model, which becomes prohibitive when a large number of downstream tasks are present. Therefore, many fine-tuning methods are proposed to learn incremental updates of pre-trained weights in a parameter efficient way, e.g., low-rank increments. These methods often evenly distribute the budget of incremental updates across all pre-trained weight matrices, and overlook the varying importance of different weight parameters. As a consequence, the fine-tuning performance is suboptimal. To bridge this gap, we propose AdaLoRA, which adaptively allocates the parameter budget among weight matrices according to their importance score. In particular, AdaLoRA parameterizes the incremental updates in the form of singular value decomposition. Such a novel approach allows us to effectively prune the singular values of unimportant updates, which is essentially to reduce their parameter budget but circumvent intensive exact SVD computations. We conduct extensive experiments with several pre-trained models on natural language processing, question answering, and natural language generation to validate the effectiveness of AdaLoRA. Results demonstrate that AdaLoRA manifests notable improvement over baselines, especially in the low budget settings. Our code is publicly available at <a class="link-external link-https" href="https://github.com/QingruZhang/AdaLoRA" rel="external noopener nofollow">this https URL</a> .

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve The paper attempts to address the issue of parameter efficiency when fine-tuning large pre-trained language models in natural language processing (NLP) tasks. Specifically, the common full-parameter fine-tuning approach consumes a significant amount of memory resources when handling numerous downstream tasks, making it impractical for real-world applications. Consequently, many studies have proposed parameter-efficient fine-tuning methods, such as low-rank incremental updates. However, these methods typically allocate the incremental update budget uniformly, ignoring the importance differences among various weight parameters, leading to suboptimal fine-tuning performance. To bridge this gap, the authors propose AdaLoRA (Adaptive Low-Rank Adaptation), a method that adaptively allocates the parameter budget based on the importance of the weight matrices. AdaLoRA parameterizes the incremental updates through Singular Value Decomposition (SVD) and dynamically adjusts the rank of each weight matrix to reduce the parameter budget for less important updates, thereby improving fine-tuning performance, especially under low-budget settings. ### Main Contributions 1. **Adaptive Parameter Budget Allocation**: AdaLoRA can dynamically adjust the parameter budget based on the importance of the weight matrices, thereby utilizing limited parameter resources more efficiently. 2. **Improved SVD Parameterization**: By parameterizing incremental updates through SVD, AdaLoRA can effectively prune unimportant singular values, reducing computational overhead. 3. **Global Budget Scheduler**: A global budget scheduler is proposed, which gradually reduces the initial high budget to the target budget, enhancing training stability and model performance. 4. **Experimental Validation**: Extensive experiments on multiple NLP tasks validate the effectiveness of AdaLoRA, particularly under low-budget settings. ### Experimental Results - **Natural Language Understanding (GLUE Benchmark)**: AdaLoRA demonstrates superior or comparable performance to existing methods across all budget levels. For instance, with a 0.3M parameter budget, AdaLoRA achieves an accuracy of 87.36% on the RTE task, 1.8% higher than the best baseline method. - **Question Answering Tasks (SQuAD v1.1 and SQuAD v2.0)**: AdaLoRA achieves significant improvements on the SQuAD v2.0 dataset, especially under low-budget settings. For example, using less than 0.1% of trainable parameters, AdaLoRA improves the F1 score on the SQuAD 2.0 dataset by 1.2%. In summary, AdaLoRA significantly enhances parameter-efficient fine-tuning performance by adaptively allocating the parameter budget, particularly in resource-constrained scenarios.

AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning

IncreLoRA: Incremental Parameter Allocation Method for Parameter-Efficient Fine-tuning

ALoRA: Allocating Low-Rank Adaptation for Fine-tuning Large Language Models

LoRA$^2$ : Multi-Scale Low-Rank Approximations for Fine-Tuning Large Language Models

ASLoRA: Adaptive Sharing Low-Rank Adaptation Across Layers

LoRA-SP: Streamlined Partial Parameter Adaptation for Resource-Efficient Fine-Tuning of Large Language Models

VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Banks

NoRA: Nested Low-Rank Adaptation for Efficient Fine-Tuning Large Models

DoRA: Enhancing Parameter-Efficient Fine-Tuning with Dynamic Rank Distribution

ShareLoRA: Parameter Efficient and Robust Large Language Model Fine-tuning via Shared Low-Rank Adaptation

Low-Rank Adaptation with Task-Relevant Feature Enhancement for Fine-tuning Language Models

Bayesian-LoRA: LoRA based Parameter Efficient Fine-Tuning using Optimal Quantization levels and Rank Values trough Differentiable Bayesian Gates

ADT: an Additive Delta-Tuning Approach for Parameter-Efficient Tuning in Pre-Trained Language Models

PRILoRA: Pruned and Rank-Increasing Low-Rank Adaptation

Matrix-Transformation Based Low-Rank Adaptation (MTLoRA): A Brain-Inspired Method for Parameter-Efficient Fine-Tuning

LoRA-GA: Low-Rank Adaptation with Gradient Approximation

Structure-Aware Low-Rank Adaptation for Parameter-Efficient Fine-Tuning

BiLoRA: A Bi-level Optimization Framework for Overfitting-Resilient Low-Rank Adaptation of Large Pre-trained Models

LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters

LoRA-Pro: Are Low-Rank Adapters Properly Optimized?