Abstract:Generative pre-trained large language models (LLMs) have demonstrated impressive performance over a wide range of tasks, thanks to the unprecedented amount of data they have been trained on. As established scaling laws indicate, LLMs' future performance improvement depends on the amount of computing and data sources they can leverage for pre-training. Federated learning (FL) has the potential to unleash the majority of the planet's data and computational resources, which are underutilized by the data-center-focused training methodology of current LLM practice. Our work presents a robust, flexible, reproducible FL approach that enables large-scale collaboration across institutions to train LLMs. We propose a scalable deployment system called Photon to enable the investigation and development of this new training paradigm for LLM pre-training. We show that Photon can be used by organizations interested in collaborating with their private data sources and computational resources for pre-training LLMs with billions of parameters. This paradigm would mobilize more computational and data resources while matching or potentially exceeding centralized performance. We further show the effectiveness of the federated training scales with model size and present our approach for training billion-scale federated LLMs using limited resources. Thus far, we have used Photon to train LLM models to the size of 7B parameters and anticipate larger models being completed in the near future. Finally, we show that LLM training is highly resilient to the classical challenges of federated statistical and hardware heterogeneity. Furthermore, we show that convergence is robust to partial participation, opening the avenue for compute-efficient collaborative training. Photon will help data-rich actors to become the protagonists of LLMs pre-training instead of leaving the stage to compute-rich actors alone.

Towards Federated Large Language Models: Motivations, Methods, and Future Directions

Federated Large Language Models: Current Progress and Future Directions

Federated Large Language Model: Solutions, Challenges and Future Directions

OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learning

Integration of Large Language Models and Federated Learning

Federated Learning driven Large Language Models for Swarm Intelligence: A Survey

FATE-LLM: A Industrial Grade Federated Learning Framework for Large Language Models

eFedLLM: Efficient LLM Inference Based on Federated Learning

Privacy-Preserving Large Language Models: Mechanisms, Applications, and Future Directions

Analysis of Privacy Leakage in Federated Large Language Models

Federated Foundation Models: Privacy-Preserving and Collaborative Learning for Large Models

FedJudge: Federated Legal Large Language Model

FedEval-LLM: Federated Evaluation of Large Language Models on Downstream Tasks with Collective Wisdom

Privacy in Large Language Models: Attacks, Defenses and Future Directions

Safely Learning with Private Data: A Federated Learning Framework for Large Language Model

Personalized Wireless Federated Learning for Large Language Models

The Future of Large Language Model Pre-training is Federated

MLLM-LLaVA-FL: Multimodal Large Language Model Assisted Federated Learning

Worldwide Federated Training of Language Models

Federated learning: A cutting-edge survey of the latest advancements and applications

LanFL: Differentially Private Federated Learning with Large Language Models using Synthetic Samples