Abstract:Low-rank adaptation (LoRA) and its variants are widely employed in fine-tuning large models, including large language models for natural language processing and diffusion models for computer vision. This paper proposes a generalized framework called SuperLoRA that unifies and extends different LoRA variants, which can be realized under different hyper-parameter settings. Introducing grouping, folding, shuffling, projecting, and tensor factoring, SuperLoRA offers high flexibility compared with other LoRA variants and demonstrates superior performance for transfer learning tasks especially in the extremely few-parameter regimes.

What problem does this paper attempt to address?

The paper primarily aims to address the issues of excessive resource consumption and high data requirements in large neural network models for downstream tasks, particularly for vision tasks (such as Vision Transformer, ConvNeXt) and natural language processing tasks (such as GPT, PALM2, Gemini, LLaMA2). To tackle these problems, the authors propose a new parameter-efficient fine-tuning framework—SuperLoRA. The goal of SuperLoRA is to unify and extend different low-rank adaptation (LoRA) variants and provide a more flexible approach to adjusting the weight updates of different attention modules. Specifically, SuperLoRA introduces mechanisms such as grouping, folding, shuffling, projection, and tensor decomposition, which can demonstrate superior transfer learning performance with an extremely small number of parameters. The key contributions of the paper are as follows: 1. **Proposing the SuperLoRA framework**: This is a new parameter-efficient fine-tuning framework that can unify and extend most LoRA variants. 2. **Parameter-efficient weight updates**: Through projected tensor rank decomposition, SuperLoRA can jointly adapt all weights across layers while providing a wide range of adjustable parameter amounts. 3. **Investigating the impact of various techniques**: Including tensor reshaping, grouping, random projection, and shuffling on performance. 4. **Empirical results**: Demonstrating high parameter efficiency of SuperLoRA on two transfer learning tasks (image classification and image generation) for large vision Transformers and diffusion models. 5. **Significant parameter reduction**: Achieving 3 to 10 times reduction in parameter amounts. Through these contributions, SuperLoRA provides a general framework for existing LoRA variants while achieving better performance and higher parameter efficiency in practical applications.

SuperLoRA: Parameter-Efficient Unified Adaptation of Multi-Layer Attention Modules

MultiLoRA: Democratizing LoRA for Better Multi-Task Learning

HyperLoRA: Efficient Cross-task Generalization Via Constrained Low-Rank Adapters Generation

ASLoRA: Adaptive Sharing Low-Rank Adaptation Across Layers

LoRA-SP: Streamlined Partial Parameter Adaptation for Resource-Efficient Fine-Tuning of Large Language Models

Enhancing Parameter Efficiency and Generalization in Large-Scale Models: A Regularized and Masked Low-Rank Adaptation Approach

Expressive and Generalizable Low-rank Adaptation for Large Models via Slow Cascaded Learning

LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters

LoRA$^2$ : Multi-Scale Low-Rank Approximations for Fine-Tuning Large Language Models

LoRA-Mini : Adaptation Matrices Decomposition and Selective Training

Less is More: Extreme Gradient Boost Rank-1 Adaption for Efficient Finetuning of LLMs

Sparse Low-rank Adaptation of Pre-trained Language Models

ResLoRA: Identity Residual Mapping in Low-Rank Adaption

CoRA: Optimizing Low-Rank Adaptation with Common Subspace of Large Language Models

SwitchLoRA: Switched Low-Rank Adaptation Can Learn Full-Rank Information

Matrix-Transformation Based Low-Rank Adaptation (MTLoRA): A Brain-Inspired Method for Parameter-Efficient Fine-Tuning

Batched Low-Rank Adaptation of Foundation Models

MTL-LoRA: Low-Rank Adaptation for Multi-Task Learning

Chain of LoRA: Efficient Fine-tuning of Language Models via Residual Learning

LoRA-Pro: Are Low-Rank Adapters Properly Optimized?

A Survey on LoRA of Large Language Models