Abstract:Transformer-based, pre-trained large language models (LLMs) have demonstrated outstanding performance across diverse domains, particularly in the emerging {\em pretrain-then-finetune} paradigm. Low-Rank Adaptation (LoRA), a parameter-efficient fine-tuning method, is commonly used to adapt a base LLM to multiple downstream tasks. Further, LLM platforms enable developers to fine-tune multiple models and develop various domain-specific applications simultaneously. However, existing model parallelism schemes suffer from high communication overhead and inefficient GPU utilization when training multiple LoRA tasks across GPUs and machines. In this paper, we present mLoRA, a parallelism-efficient fine-tuning system designed for training multiple LoRA across GPUs and machines. mLoRA introduces a novel LoRA-aware pipeline parallelism scheme that efficiently pipelines independent LoRA adapters and their distinct fine-tuning stages across GPUs and machines, along with a new LoRA-efficient operator to enhance GPU utilization during pipelined LoRA training. Our extensive evaluation shows that mLoRA can significantly reduce average fine-tuning task completion time, e.g., by 30\%, compared to state-of-the-art methods like FSDP. More importantly, mLoRA enables simultaneous fine-tuning of larger models, e.g., two Llama-2-13B models on four NVIDIA RTX A6000 48GB GPUs, which is not feasible for FSDP due to high memory requirements. Hence, mLoRA not only increases fine-tuning efficiency but also makes it more accessible on cost-effective GPUs. mLoRA has been deployed in AntGroup's production environment.

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is the high communication overhead and inefficient GPU utilization when existing model parallelism schemes train multiple LoRA (Low - Rank Adaptation) adapters on multiple GPUs. Specifically: 1. **High communication overhead**: Existing model parallel methods (such as tensor parallelism and pipeline parallelism) require a large amount of communication overhead when synchronizing across GPUs or across machines, especially when dealing with large - scale language models (LLMs), which will significantly slow down the training speed. 2. **Low GPU utilization**: Due to the dependencies in pipeline parallelism, the GPU idle time increases, especially when dealing with multiple small - scale LoRA adapters. Frequent GPU kernel launches increase the total training time. 3. **Memory limitations**: When multiple LoRA adapters need to be fine - tuned simultaneously, the memory of a single GPU is often insufficient to support such an operation. Therefore, multiple GPUs are required for distributed training. However, existing parallelization methods cannot effectively utilize multi - GPU resources when dealing with large models (such as Llama - 2 - 13B) due to high memory requirements. To solve these problems, the authors propose mLoRA, an efficient fine - tuning system, which aims to improve the fine - tuning efficiency of LoRA adapters in a multi - GPU environment in the following ways: - **Introducing a new pipeline parallel mechanism (LoRAPP)**: By independently scheduling the forward and backward propagation stages of different LoRA adapters, the "bubble" problem (i.e., GPU idle time) in traditional pipeline parallelism is avoided, thereby improving GPU utilization. - **Developing a new efficient operator (BatchLoRA)**: Multiple LoRA fine - tuning tasks are combined into one large batch, and matrix multiplication operations are collectively executed, reducing the number of kernel launches and further improving GPU utilization. Through these improvements, mLoRA can significantly reduce the completion time of fine - tuning tasks in a multi - GPU environment and can simultaneously fine - tune larger - scale models. For example, two Llama - 2 - 13B models can be simultaneously fine - tuned on four NVIDIA RTX A6000 48GB GPUs. In addition, mLoRA also reduces the demand for high - end GPUs, making fine - tuning more cost - effective. In summary, the main contribution of this paper is to provide an efficient solution to the performance bottleneck problems encountered by existing methods when training multiple LoRA adapters in a multi - GPU environment, thereby improving the fine - tuning efficiency and reducing the cost.

mLoRA: Fine-Tuning LoRA Adapters via Highly-Efficient Pipeline Parallelism in Multiple GPUs

LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Report

S-LoRA: Serving Thousands of Concurrent LoRA Adapters

MultiLoRA: Democratizing LoRA for Better Multi-Task Learning

MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuning

MeteoRA: Multiple-tasks Embedded LoRA for Large Language Models

DLP-LoRA: Efficient Task-Specific LoRA Fusion with a Dynamic, Lightweight Plugin for Large Language Models

MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts

PeriodicLoRA: Breaking the Low-Rank Bottleneck in LoRA Optimization

V-LoRA: an Efficient and Flexible System Boosts Vision Applications with LoRA LMM

Matrix-Transformation Based Low-Rank Adaptation (MTLoRA): A Brain-Inspired Method for Parameter-Efficient Fine-Tuning

A Framework to Implement 1+N Multi-task Fine-tuning Pattern in LLMs Using the CGC-LORA Algorithm

LoRA-GA: Low-Rank Adaptation with Gradient Approximation

MALoRA: Mixture of Asymmetric Low-Rank Adaptation for Enhanced Multi-Task Learning

MTL-LoRA: Low-Rank Adaptation for Multi-Task Learning

LoRA-Pro: Are Low-Rank Adapters Properly Optimized?

LoRA$^2$ : Multi-Scale Low-Rank Approximations for Fine-Tuning Large Language Models

LoRA-SP: Streamlined Partial Parameter Adaptation for Resource-Efficient Fine-Tuning of Large Language Models

LaMDA: Large Model Fine-Tuning via Spectrally Decomposed Low-Dimensional Adaptation

Mixture-of-LoRAs: An Efficient Multitask Tuning for Large Language Models

ALoRA: Allocating Low-Rank Adaptation for Fine-tuning Large Language Models