Abstract:Fine-tuning large-scale pretrained models is prohibitively expensive in terms of computational and memory costs. LoRA, as one of the most popular Parameter-Efficient Fine-Tuning (PEFT) methods, offers a cost-effective alternative by fine-tuning an auxiliary low-rank model that has significantly fewer parameters. Although LoRA reduces the computational and memory requirements significantly at each iteration, extensive empirical evidence indicates that it converges at a considerably slower rate compared to full fine-tuning, ultimately leading to increased overall compute and often worse test performance. In our paper, we perform an in-depth investigation of the initialization method of LoRA and show that careful initialization (without any change of the architecture and the training algorithm) can significantly enhance both efficiency and performance. In particular, we introduce a novel initialization method, LoRA-GA (Low Rank Adaptation with Gradient Approximation), which aligns the gradients of low-rank matrix product with those of full fine-tuning at the first step. Our extensive experiments demonstrate that LoRA-GA achieves a convergence rate comparable to that of full fine-tuning (hence being significantly faster than vanilla LoRA as well as various recent improvements) while simultaneously attaining comparable or even better performance. For example, on the subset of the GLUE dataset with T5-Base, LoRA-GA outperforms LoRA by 5.69% on average. On larger models such as Llama 2-7B, LoRA-GA shows performance improvements of 0.34, 11.52%, and 5.05% on MT-bench, GSM8K, and Human-eval, respectively. Additionally, we observe up to 2-4 times convergence speed improvement compared to vanilla LoRA, validating its effectiveness in accelerating convergence and enhancing model performance. Code is available at <a class="link-external link-https" href="https://github.com/Outsider565/LoRA-GA" rel="external noopener nofollow">this https URL</a>.

Training Neural Networks from Scratch with Parallel Low-Rank Adapters

HyperLoRA: Efficient Cross-task Generalization Via Constrained Low-Rank Adapters Generation

Computational Limits of Low-Rank Adaptation (LoRA) for Transformer-Based Models

Riemannian Preconditioned LoRA for Fine-Tuning Foundation Models

LoRA+: Efficient Low Rank Adaptation of Large Models

LoRA-Mini : Adaptation Matrices Decomposition and Selective Training

Learning on LoRAs: GL-Equivariant Processing of Low-Rank Weight Spaces for Large Finetuned Models

NoRA: Nested Low-Rank Adaptation for Efficient Fine-Tuning Large Models

LoRTA: Low Rank Tensor Adaptation of Large Language Models

The Expressive Power of Low-Rank Adaptation

LoRA-GA: Low-Rank Adaptation with Gradient Approximation

An Empirical Analysis of Forgetting in Pre-trained Models with Incremental Low-Rank Updates

NEAT: Nonlinear Parameter-efficient Adaptation of Pre-trained Models

LoRA-Pro: Are Low-Rank Adapters Properly Optimized?

Batched Low-Rank Adaptation of Foundation Models

GeoLoRA: Geometric integration for parameter efficient fine-tuning

mLoRA: Fine-Tuning LoRA Adapters via Highly-Efficient Pipeline Parallelism in Multiple GPUs

Flat-LoRA: Low-Rank Adaption over a Flat Loss Landscape

PRILoRA: Pruned and Rank-Increasing Low-Rank Adaptation

Dual Low-Rank Adaptation for Continual Learning with Pre-Trained Models

Run LoRA Run: Faster and Lighter LoRA Implementations