Abstract:Low-rank adaptations (LoRA) are often employed to fine-tune large language models (LLMs) for new tasks. This paper investigates LoRA composability for cross-task generalization and introduces LoraHub, a simple framework devised for the purposive assembly of LoRA modules trained on diverse given tasks, with the objective of achieving adaptable performance on unseen tasks. With just a few examples from a new task, LoraHub can fluidly combine multiple LoRA modules, eliminating the need for human expertise and assumptions. Notably, the composition requires neither additional model parameters nor gradients. Empirical results on the Big-Bench Hard benchmark suggest that LoraHub, while not surpassing the performance of in-context learning, offers a notable performance-efficiency trade-off in few-shot scenarios by employing a significantly reduced number of tokens per example during inference. Notably, LoraHub establishes a better upper bound compared to in-context learning when paired with different demonstration examples, demonstrating its potential for future development. Our vision is to establish a platform for LoRA modules, empowering users to share their trained LoRA modules. This collaborative approach facilitates the seamless application of LoRA modules to novel tasks, contributing to an adaptive ecosystem. Our code is available at <a class="link-external link-https" href="https://github.com/sail-sg/lorahub" rel="external noopener nofollow">this https URL</a>, and all the pre-trained LoRA modules are released at <a class="link-external link-https" href="https://huggingface.co/lorahub" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to achieve cross - task generalization by combining Low - Rank Adaptation (LoRA) modules in the few - shot scenario. Specifically, the paper proposes a method named LoraHub, which can efficiently adapt to unseen new tasks by dynamically combining LoRA modules trained on different tasks. This method aims to reduce the dependence on human expert knowledge and does not require additional model parameters or gradient updates, thereby significantly reducing the computational cost and memory usage while maintaining high performance. The main contributions of LoraHub are as follows: 1. **Cross - task generalization**: By combining multiple LoRA modules, LoraHub can quickly adapt to new tasks with only a small number of new task examples. 2. **No additional parameters or gradients**: The combination process of LoraHub does not increase model parameters and does not require gradient updates, thus reducing the computational complexity. 3. **Automatic combination**: LoraHub can automatically select and combine LoRA modules, reducing the need for human intervention. 4. **Balance between performance and efficiency**: In the few - shot scenario, the performance of LoraHub is close to that of in - context learning, but the number of tokens used is significantly reduced, thus achieving a good balance between performance and efficiency. The paper verifies the effectiveness of LoraHub through experiments on the Big - Bench Hard benchmark. The results show that LoraHub performs better than zero - shot learning in the few - shot scenario and even outperforms in - context learning on some tasks. In addition, LoraHub also demonstrates its economic and efficiency advantages in handling repetitive tasks.

LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition

HyperLoRA: Efficient Cross-task Generalization Via Constrained Low-Rank Adapters Generation

Instance-Level Dynamic LoRAs Composition for Cross-Task Generalization

LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wild

MultiLoRA: Democratizing LoRA for Better Multi-Task Learning

SuperLoRA: Parameter-Efficient Unified Adaptation of Multi-Layer Attention Modules

Merging LoRAs like Playing LEGO: Pushing the Modularity of LoRA to Extremes Through Rank-Wise Clustering

DLP-LoRA: Efficient Task-Specific LoRA Fusion with a Dynamic, Lightweight Plugin for Large Language Models

A Note on LoRA

LoRA-Flow: Dynamic LoRA Fusion for Large Language Models in Generative Tasks

Chain-of-LoRA: Enhancing the Instruction Fine-Tuning Performance of Low-Rank Adaptation on Diverse Instruction Set

Batched Low-Rank Adaptation of Foundation Models

CA-LoRA: Adapting Existing LoRA for Compressed LLMs to Enable Efficient Multi-Tasking on Personal Devices

Unlocking the Global Synergies in Low-Rank Adapters

LoraMap: Harnessing the Power of LoRA Connections

LoRA Soups: Merging LoRAs for Practical Skill Composition Tasks

Personalized LoRA for Human-Centered Text Understanding

Retrieval-Augmented Mixture of LoRA Experts for Uploadable Machine Learning

V-LoRA: an Efficient and Flexible System Boosts Vision Applications with LoRA LMM

Dlora: Dynamically Orchestrating Requests and Adapters for LoRA LLM Serving.

A Survey on LoRA of Large Language Models