Sparse Low-rank Adaptation of Pre-trained Language Models

Ning Ding,Xingtai Lv,Qiaosen Wang,Yulin Chen,Bowen Zhou,Zhiyuan Liu,Maosong Sun

2023-11-20

Abstract:Fine-tuning pre-trained large language models in a parameter-efficient manner is widely studied for its effectiveness and efficiency. The popular method of low-rank adaptation (LoRA) offers a notable approach, hypothesizing that the adaptation process is intrinsically low-dimensional. Although LoRA has demonstrated commendable performance, it is implemented with a fixed and unalterable intrinsic rank that might not always be the ideal choice. Recognizing the need for more flexible adaptation, we extend the methodology of LoRA to an innovative approach we call sparse low-rank adaptation (SoRA) that enables dynamic adjustments to the intrinsic rank during the adaptation process. We achieve this through the incorporation of a gate unit optimized with proximal gradient method in the training stage, controlling the cardinality of rank under the sparsity of the gate. In the subsequent inference stage, we eliminate the parameter blocks corresponding to the zeroed-out ranks, to reduce each SoRA module back to a concise yet rank-optimal LoRA. Our approach strengthens the representation power of LoRA by initializing it with a higher rank, while efficiently taming a temporarily increased number of parameters via updating in a sparse way. We further introduce a sparsifying scheduler for SoRA, aiming to examine the impact of the number of non-zero parameters on the model's memorization and generalization. Our experimental results demonstrate that SoRA can outperform other baselines even with 70% retained parameters and 70% training time.

Computation and Language,Artificial Intelligence,Machine Learning

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to flexibly adjust the intrinsic rank during the adaptation process when performing parameter - efficient fine - tuning of pre - trained language models. Although the traditional low - rank adaptation method (LoRA) performs well, its intrinsic rank is a fixed hyper - parameter and may not be the best choice in all cases. Therefore, this paper proposes a new method - Sparse Low - Rank Adaptation (SoRA), which aims to dynamically adjust the intrinsic rank by introducing a gating unit optimized by the proximal gradient method during the training stage, thereby improving the representational ability and adaptation efficiency of the model. Specifically, SoRA allows the use of a higher rank at initialization and effectively manages the temporarily increased number of parameters through sparse updates. In addition, the authors also introduce a sparsification scheduler to study the impact of the number of non - zero parameters on the model's memory and generalization ability. Experimental results show that even when only 70% of the parameters and 70% of the training time are retained, SoRA can still outperform other baseline methods.