Abstract:Private data, being larger and quality-higher than public data, can greatly improve large language models (LLM). However, due to privacy concerns, this data is often dispersed in multiple silos, making its secure utilization for LLM training a challenge. Federated learning (FL) is an ideal solution for training models with distributed private data, but traditional frameworks like FedAvg are unsuitable for LLM due to their high computational demands on clients. An alternative, split learning, offloads most training parameters to the server while training embedding and output layers locally, making it more suitable for LLM. Nonetheless, it faces significant challenges in security and efficiency. Firstly, the gradients of embeddings are prone to attacks, leading to potential reverse engineering of private data. Furthermore, the server's limitation of handle only one client's training request at a time hinders parallel training, severely impacting training efficiency. In this paper, we propose a Federated Learning framework for LLM, named FL-GLM, which prevents data leakage caused by both server-side and peer-client attacks while improving training efficiency. Specifically, we first place the input block and output block on local client to prevent embedding gradient attacks from server. Secondly, we employ key-encryption during client-server communication to prevent reverse engineering attacks from peer-clients. Lastly, we employ optimization methods like client-batching or server-hierarchical, adopting different acceleration methods based on the actual computational capabilities of the server. Experimental results on NLU and generation tasks demonstrate that FL-GLM achieves comparable metrics to centralized chatGLM model, validating the effectiveness of our federated learning framework.

DisLLM: Distributed LLMs for Privacy Assurance in Resource-Constrained Environments

eFedLLM: Efficient LLM Inference Based on Federated Learning

OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learning

FedJudge: Federated Legal Large Language Model

Towards Federated Large Language Models: Motivations, Methods, and Future Directions

Differentially Private Low-Rank Adaptation of Large Language Model Using Federated Learning

LLMs as On-demand Customizable Service

LanFL: Differentially Private Federated Learning with Large Language Models using Synthetic Samples

Federated Large Language Models: Current Progress and Future Directions

Split-and-Denoise: Protect large language model inference with local differential privacy

FwdLLM: Efficient FedLLM using Forward Gradient

Federated Large Language Model: Solutions, Challenges and Future Directions

Safely Learning with Private Data: A Federated Learning Framework for Large Language Model

CELLM: An Efficient Communication in Large Language Models Training for Federated Learning

FedSpaLLM: Federated Pruning of Large Language Models

A Fast, Performant, Secure Distributed Training Framework For Large Language Model

Analysis of Privacy Leakage in Federated Large Language Models

Personalized Wireless Federated Learning for Large Language Models

ScaleLLM: A Resource-Frugal LLM Serving Framework by Optimizing End-to-End Efficiency

DrLLM: Prompt-Enhanced Distributed Denial-of-Service Resistance Method with Large Language Models

DISC-LawLLM: Fine-tuning Large Language Models for Intelligent Legal Services