Abstract:Private data, being larger and quality-higher than public data, can greatly improve large language models (LLM). However, due to privacy concerns, this data is often dispersed in multiple silos, making its secure utilization for LLM training a challenge. Federated learning (FL) is an ideal solution for training models with distributed private data, but traditional frameworks like FedAvg are unsuitable for LLM due to their high computational demands on clients. An alternative, split learning, offloads most training parameters to the server while training embedding and output layers locally, making it more suitable for LLM. Nonetheless, it faces significant challenges in security and efficiency. Firstly, the gradients of embeddings are prone to attacks, leading to potential reverse engineering of private data. Furthermore, the server's limitation of handle only one client's training request at a time hinders parallel training, severely impacting training efficiency. In this paper, we propose a Federated Learning framework for LLM, named FL-GLM, which prevents data leakage caused by both server-side and peer-client attacks while improving training efficiency. Specifically, we first place the input block and output block on local client to prevent embedding gradient attacks from server. Secondly, we employ key-encryption during client-server communication to prevent reverse engineering attacks from peer-clients. Lastly, we employ optimization methods like client-batching or server-hierarchical, adopting different acceleration methods based on the actual computational capabilities of the server. Experimental results on NLU and generation tasks demonstrate that FL-GLM achieves comparable metrics to centralized chatGLM model, validating the effectiveness of our federated learning framework.

Billion-scale Federated Learning on Mobile Clients

Secure Federated Submodel Learning

Decentralized Personalized Online Federated Learning

Practical and Light-weight Secure Aggregation for Federated Submodel Learning

Federated Learning Scheme Based on Secure Multi-party Computation and Differential Privacy

DistFL: Distribution-aware Federated Learning for Mobile Scenarios

A Lightweight and Accuracy-Lossless Privacy-Preserving Method in Federated Learning

Towards Efficient Federated Learning Framework Via Selective Aggregation of Models

Safely Learning with Private Data: A Federated Learning Framework for Large Language Model

Prioritized Multi-Criteria Federated Learning

Communication Efficient Federated Learning With Heterogeneous Structured Client Models

From distributed machine learning to federated learning: In the view of data privacy and security

FAIR: Quality-Aware Federated Learning with Precise User Incentive and Model Aggregation.

Auction Based Clustered Federated Learning in Mobile Edge Computing System

Exploring System-Heterogeneous Federated Learning with Dynamic Model Selection

A Secure and Fair Federated Learning Framework Based on Consensus Incentive Mechanism

Evaluation Framework For Large-scale Federated Learning

Efficient-FedRec: Efficient Federated Learning Framework for Privacy-Preserving News Recommendation

Toward Resource-Efficient Federated Learning in Mobile Edge Computing

An Online Learning Approach for Client Selection in Federated Edge Learning under Budget Constraint

Auction-Based Cluster Federated Learning in Mobile Edge Computing Systems