Abstract:While Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA have effectively addressed GPU memory constraints during fine-tuning, their performance often falls short, especially in multidimensional task scenarios. To address this issue, one straightforward solution is to introduce task-specific LoRA modules as domain experts, leveraging the modeling of multiple experts' capabilities and thus enhancing the general capability of multi-task learning. Despite promising, these additional components often add complexity to the training and inference process, contravening the efficient characterization of PEFT designed for. Considering this, we introduce an innovative PEFT method, TeamLoRA, consisting of a collaboration and competition module for experts, and thus achieving the right balance of effectiveness and efficiency: (i) For collaboration, a novel knowledge-sharing and -organizing mechanism is devised to appropriately reduce the scale of matrix operations, thereby boosting the training and inference speed. (ii) For competition, we propose leveraging a game-theoretic interaction mechanism for experts, encouraging experts to transfer their domain-specific knowledge while facing diverse downstream tasks, and thus enhancing the performance. By doing so, TeamLoRA elegantly connects the experts as a "Team" with internal collaboration and competition, enabling a faster and more accurate PEFT paradigm for multi-task learning. To validate the superiority of TeamLoRA, we curate a comprehensive multi-task evaluation(CME) benchmark to thoroughly assess the capability of multi-task learning. Experiments conducted on our CME and other benchmarks indicate the effectiveness and efficiency of TeamLoRA. Our project is available at <a class="link-external link-https" href="https://github.com/Lin-Tianwei/TeamLoRA" rel="external noopener nofollow">this https URL</a>.

A Framework to Implement 1+N Multi-task Fine-tuning Pattern in LLMs Using the CGC-LORA Algorithm

Mixture-of-LoRAs: An Efficient Multitask Tuning for Large Language Models

mLoRA: Fine-Tuning LoRA Adapters via Highly-Efficient Pipeline Parallelism in Multiple GPUs

MultiLoRA: Democratizing LoRA for Better Multi-Task Learning

MTL-LoRA: Low-Rank Adaptation for Multi-Task Learning

Multimodal Instruction Tuning with Conditional Mixture of LoRA

HyperLoRA: Efficient Cross-task Generalization Via Constrained Low-Rank Adapters Generation

When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applications

MoDULA: Mixture of Domain-Specific and Universal LoRA for Multi-Task Learning

MIRA: A Method of Federated MultI-Task Learning for LaRge LAnguage Models

Chain-of-LoRA: Enhancing the Instruction Fine-Tuning Performance of Low-Rank Adaptation on Diverse Instruction Set

MeteoRA: Multiple-tasks Embedded LoRA for Large Language Models

LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Report

Matrix-Transformation Based Low-Rank Adaptation (MTLoRA): A Brain-Inspired Method for Parameter-Efficient Fine-Tuning

Multi-Task Instruction Tuning of LLaMa for Specific Scenarios: A Preliminary Study on Writing Assistance

Less is More: Extreme Gradient Boost Rank-1 Adaption for Efficient Finetuning of LLMs

Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning

DLP-LoRA: Efficient Task-Specific LoRA Fusion with a Dynamic, Lightweight Plugin for Large Language Models

TeamLoRA: Boosting Low-Rank Adaptation with Expert Collaboration and Competition

SuperLoRA: Parameter-Efficient Unified Adaptation of Multi-Layer Attention Modules

GraphLoRA: Empowering LLMs Fine-Tuning via Graph Collaboration of MoE