Abstract:LLMs have demonstrated great capabilities in various NLP tasks. Different entities can further improve the performance of those LLMs on their specific downstream tasks by fine-tuning LLMs. When several entities have similar interested tasks, but their data cannot be shared because of privacy concerns regulations, federated learning (FL) is a mainstream solution to leverage the data of different entities. However, fine-tuning LLMs in federated learning settings still lacks adequate support from existing FL frameworks because it has to deal with optimizing the consumption of significant communication and computational resources, data preparation for different tasks, and distinct information protection demands. This paper first discusses these challenges of federated fine-tuning LLMs, and introduces our package FS-LLM as a main contribution, which consists of the following components: (1) we build an end-to-end benchmarking pipeline, automizing the processes of dataset preprocessing, federated fine-tuning execution, and performance evaluation on federated LLM fine-tuning; (2) we provide comprehensive federated parameter-efficient fine-tuning algorithm implementations and versatile programming interfaces for future extension in FL scenarios with low communication and computation costs, even without accessing the full model; (3) we adopt several accelerating and resource-efficient operators for fine-tuning LLMs with limited resources and the flexible pluggable sub-routines for interdisciplinary study. We conduct extensive experiments to validate the effectiveness of FS-LLM and benchmark advanced LLMs with state-of-the-art parameter-efficient fine-tuning algorithms in FL settings, which also yields valuable insights into federated fine-tuning LLMs for the research community. To facilitate further research and adoption, we release FS-LLM at <a class="link-external link-https" href="https://github.com/alibaba/FederatedScope/tree/llm" rel="external noopener nofollow">this https URL</a>.

FwdLLM: Efficient Federated Finetuning of Large Language Models with Perturbed Inferences.

FwdLLM: Efficient FedLLM using Forward Gradient

eFedLLM: Efficient LLM Inference Based on Federated Learning

Personalized Wireless Federated Learning for Large Language Models

Federated Large Language Model: Solutions, Challenges and Future Directions

Federated Learning of Large Language Models with Parameter-Efficient Prompt Tuning and Adaptive Optimization

Federated Large Language Models: Current Progress and Future Directions

OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learning

On the Convergence of Zeroth-Order Federated Tuning for Large Language Models

Low-Parameter Federated Learning with Large Language Models

FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning

CG-FedLLM: How to Compress Gradients in Federated Fune-tuning for Large Language Models

FedEval-LLM: Federated Evaluation of Large Language Models on Downstream Tasks with Collective Wisdom

CELLM: An Efficient Communication in Large Language Models Training for Federated Learning

Towards Federated Large Language Models: Motivations, Methods, and Future Directions

FedPT: Federated Proxy-Tuning of Large Language Models on Resource-Constrained Edge Devices

FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations

Fisher Information-based Efficient Curriculum Federated Learning with Large Language Models

FedJudge: Federated Legal Large Language Model

FATE-LLM: A Industrial Grade Federated Learning Framework for Large Language Models

FedCoLLM: A Parameter-Efficient Federated Co-tuning Framework for Large and Small Language Models