Abstract:The rapid advancement of foundation modelslarge-scale neural networks trained on diverse, extensive datasetshas revolutionized artificial intelligence, enabling unprecedented advancements across domains such as natural language processing, computer vision, and scientific discovery. However, the substantial parameter count of these models, often reaching billions or trillions, poses significant challenges in adapting them to specific downstream tasks. Low-Rank Adaptation (LoRA) has emerged as a highly promising approach for mitigating these challenges, offering a parameter-efficient mechanism to fine-tune foundation models with minimal computational overhead. This survey provides the first comprehensive review of LoRA techniques beyond large Language Models to general foundation models, including recent techniques foundations, emerging frontiers and applications of low-rank adaptation across multiple domains. Finally, this survey discusses key challenges and future research directions in theoretical understanding, scalability, and robustness. This survey serves as a valuable resource for researchers and practitioners working with efficient foundation model adaptation.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: How to efficiently adapt to specific downstream tasks in large - scale foundation models (Foundation Models), especially for models with a huge number of parameters (such as models with billions or trillions of parameters). The traditional full - parameter fine - tuning method becomes impractical due to the excessive demand for computing resources. Low - Rank Adaptation (LoRA), as a promising method, aims to fine - tune these foundation models through a parameter - efficient mechanism, thereby achieving task adaptation with minimal computational overhead. ### Specific Problems and Challenges 1. **Computational and Storage Costs of Large - scale Models**: The number of parameters in foundation models is huge, resulting in very high computational and storage costs when adapting to specific downstream tasks. 2. **Limitations of Traditional Fine - tuning Methods**: Traditional fine - tuning methods involve updating all model parameters. For large - scale foundation models, this not only has high computational complexity but also is prone to overfitting and catastrophic forgetting. 3. **Requirements for Multi - task Learning and Continuous Learning**: A method is needed that can flexibly switch between multiple tasks without affecting the basic performance of the model. ### Solutions of Low - Rank Adaptation (LoRA) - **Parameter Efficiency**: By decomposing the weight update matrix into low - rank matrices, the number of parameters that need to be trained is greatly reduced, and the computational and storage requirements are lowered. - **Modular Adaptation**: It allows the creation of lightweight task - specific adapters that can be exchanged without modifying the basic model architecture, supporting multi - task learning and task switching. - **Knowledge Retention**: By freezing the pre - trained weights, catastrophic forgetting is avoided while obtaining task - specific capabilities. - **Flexible Deployment**: The compact adaptation method enables the model to be flexibly combined or alternately used between different tasks or domains, improving the flexibility of deployment. ### Main Contributions of the Paper 1. **Systematic Analysis of Technical Foundations**: A structured analysis of the technical progress of LoRA is carried out, including parameter - efficiency strategies, rank - adaptation mechanisms, improvements in the training process, and emerging theoretical perspectives. 2. **Emerging Frontier Research**: Advanced architectures (such as multi - LoRA combinations and expert - mixing methods), continuous learning, federated learning, and other emerging research directions are explored. 3. **Application Review**: A comprehensive review of the practical applications of LoRA in multiple fields such as natural language processing, computer vision, speech recognition, and scientific discovery is presented. Through these contributions, the paper provides a valuable resource for researchers and practitioners, helping them better understand and apply low - rank adaptation techniques and promoting the further development of foundation models in various fields.

Low-Rank Adaptation for Foundation Models: A Comprehensive Review

A Survey on LoRA of Large Language Models

Batched Low-Rank Adaptation of Foundation Models

The Expressive Power of Low-Rank Adaptation

On Fairness of Low-Rank Adaptation of Large Models

Low-Rank Adaptation for Multilingual Summarization: An Empirical Study

A concise analysis of low-rank adaptation

Randomized Asymmetric Chain of LoRA: The First Meaningful Theoretical Framework for Low-Rank Adaptation

Expressive and Generalizable Low-rank Adaptation for Large Models via Slow Cascaded Learning

Chain of LoRA: Efficient Fine-tuning of Language Models via Residual Learning

Structure-Aware Low-Rank Adaptation for Parameter-Efficient Fine-Tuning

Enhancing Parameter Efficiency and Generalization in Large-Scale Models: A Regularized and Masked Low-Rank Adaptation Approach

Less is More: Extreme Gradient Boost Rank-1 Adaption for Efficient Finetuning of LLMs

Sparse Low-rank Adaptation of Pre-trained Language Models

LoRA-Pro: Are Low-Rank Adapters Properly Optimized?

NoRA: Nested Low-Rank Adaptation for Efficient Fine-Tuning Large Models

Computational Limits of Low-Rank Adaptation (LoRA) for Transformer-Based Models

ALoRA: Allocating Low-Rank Adaptation for Fine-tuning Large Language Models

LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters

LoRA-Mini : Adaptation Matrices Decomposition and Selective Training