Abstract:The scalability of large language models (LLMs) in handling high-complexity models and large-scale datasets has led to tremendous successes in pivotal domains. While there is an urgent need to acquire more training data for LLMs, a concerning reality is the depletion of high-quality public datasets within a few years. In view of this, the federated learning (FL) LLM fine-tuning paradigm recently has been proposed to facilitate collaborative LLM fine-tuning on distributed private data, where multiple data owners collaboratively fine-tune a shared LLM without sharing raw data. However, the staggering model size of LLMs imposes heavy computing and communication burdens on clients, posing significant barriers to the democratization of the FL LLM fine-tuning paradigm. To address this issue, split learning (SL) has emerged as a promising solution by offloading the primary training workload to a server via model partitioning while exchanging activation/activation's gradients with smaller data sizes rather than the entire LLM. Unfortunately, research on the SL LLM fine-tuning paradigm is still in its nascent stage. To fill this gap, in this paper, we propose the first SL LLM fine-tuning framework, named SplitLoRA. SplitLoRA is built on the split federated learning (SFL) framework, amalgamating the advantages of parallel training from FL and model splitting from SL and thus greatly enhancing the training efficiency. It is worth noting that SplitLoRA is the inaugural open-source benchmark for SL LLM fine-tuning, providing a foundation for research efforts dedicated to advancing SL LLM fine-tuning. Extensive simulations validate that SplitLoRA achieves target accuracy in significantly less time than state-of-the-art LLM fine-tuning frameworks, demonstrating the superior training performance of SplitLoRA. The project page is available at <a class="link-external link-https" href="https://fduinc.github.io/splitlora/" rel="external noopener nofollow">this https URL</a>.

Exploring Gradient Subspaces: Addressing and Overcoming LoRA's Limitations in Federated Fine-Tuning of Large Language Models

FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations

Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resources

Improving LoRA in Privacy-preserving Federated Learning

Federated LLMs Fine-tuned with Adaptive Importance-Aware LoRA

Towards Robust and Efficient Federated Low-Rank Adaptation with Heterogeneous Clients

FDLoRA: Personalized Federated Learning of Large Language Model via Dual LoRA Tuning

Federated LoRA with Sparse Communication

Differentially Private Low-Rank Adaptation of Large Language Model Using Federated Learning

LoRA-FAIR: Federated LoRA Fine-Tuning with Aggregation and Initialization Refinement

Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Models

Learning on LoRAs: GL-Equivariant Processing of Low-Rank Weight Spaces for Large Finetuned Models

Federated Low-Rank Adaptation for Large Models Fine-Tuning over Wireless Networks

Selective Aggregation for Low-Rank Adaptation in Federated Learning

RBLA: Rank-Based-LoRA-Aggregation for Fine-tuning Heterogeneous Models in FLaaS

Exact Aggregation for Federated and Efficient Fine-Tuning of Foundation Models

Less is More: Extreme Gradient Boost Rank-1 Adaption for Efficient Finetuning of LLMs

SA-FedLora: Adaptive Parameter Allocation for Efficient Federated Learning with LoRA Tuning

SplitLoRA: A Split Parameter-Efficient Fine-Tuning Framework for Large Language Models

GeLoRA: Geometric Adaptive Ranks For Efficient LoRA Fine-tuning

LoRA vs Full Fine-tuning: An Illusion of Equivalence