Abstract:The scalability of large language models (LLMs) in handling high-complexity models and large-scale datasets has led to tremendous successes in pivotal domains. While there is an urgent need to acquire more training data for LLMs, a concerning reality is the depletion of high-quality public datasets within a few years. In view of this, the federated learning (FL) LLM fine-tuning paradigm recently has been proposed to facilitate collaborative LLM fine-tuning on distributed private data, where multiple data owners collaboratively fine-tune a shared LLM without sharing raw data. However, the staggering model size of LLMs imposes heavy computing and communication burdens on clients, posing significant barriers to the democratization of the FL LLM fine-tuning paradigm. To address this issue, split learning (SL) has emerged as a promising solution by offloading the primary training workload to a server via model partitioning while exchanging activation/activation's gradients with smaller data sizes rather than the entire LLM. Unfortunately, research on the SL LLM fine-tuning paradigm is still in its nascent stage. To fill this gap, in this paper, we propose the first SL LLM fine-tuning framework, named SplitLoRA. SplitLoRA is built on the split federated learning (SFL) framework, amalgamating the advantages of parallel training from FL and model splitting from SL and thus greatly enhancing the training efficiency. It is worth noting that SplitLoRA is the inaugural open-source benchmark for SL LLM fine-tuning, providing a foundation for research efforts dedicated to advancing SL LLM fine-tuning. Extensive simulations validate that SplitLoRA achieves target accuracy in significantly less time than state-of-the-art LLM fine-tuning frameworks, demonstrating the superior training performance of SplitLoRA. The project page is available at <a class="link-external link-https" href="https://fduinc.github.io/splitlora/" rel="external noopener nofollow">this https URL</a>.

LoRa Modulation for Split Learning

Semantics Alignment via Split Learning for Resilient Multi-User Semantic Communication

SplitAMC: Split Learning for Robust Automatic Modulation Classification

Adaptive Modulation for Wireless Federated Learning

NELoRa: Towards Ultra-low SNR LoRa Communication with Neural-enhanced Demodulation

SplitLoRA: A Split Parameter-Efficient Fine-Tuning Framework for Large Language Models

SplitMAC: Wireless Split Learning over Multiple Access Channels

Adaptive Modulation Scheme for Satellite Communication Channel Based on RLNN

FedsLLM: Federated Split Learning for Large Language Models over Communication Networks

Communication-Efficient Split Learning via Adaptive Feature-Wise Compression

Over-the-Air Split Learning with MIMO-Based Neural Network and Constellation-Based Activation

End-to-End Learning for SLP-Based ISAC Systems

Split Learning for Sensing-Aided Single and Multi-Level Beam Selection in Multi-Vendor RAN

Federated LoRA with Sparse Communication

NELoRa-Bench: A Benchmark for Neural-enhanced LoRa Demodulation

Split Learning Over Wireless Networks: Parallel Design and Resource Management

Accelerating Split Federated Learning over Wireless Communication Networks

End-to-End Learning for Uplink MU-SIMO Joint Transmitter and Non-Coherent Receiver Design in Fading Channels.

Learning to Modulate for Non-coherent MIMO

Machine-Learning-Assisted Transmission Power Control for LoRaWAN Considering Environments With High Signal- to -Noise Variation

Deep Learning-Based Automatic Modulation Recognition in OTFS and OFDM Systems.