Abstract:The recent advancement of large foundation models (FMs) has increased the demand for fine-tuning these models on large-scale and cross-domain datasets. To address this, federated fine-tuning has emerged as a solution, allowing models to be fine-tuned on distributed datasets across multiple devices while ensuring data privacy. However, the substantial parameter size of FMs and the multi-round communication required by traditional federated fine-tuning algorithms result in prohibitively high communication costs, challenging the practicality of federated fine-tuning. In this paper, we are the first to reveal, both theoretically and empirically, that the traditional multi-round aggregation algorithms may not be necessary for federated fine-tuning large FMs. Our experiments reveal that a single round of communication (i.e., one-shot federated fine-tuning) yields a global model performance comparable to that achieved through multiple rounds of communication. Through rigorous mathematical and empirical analyses, we demonstrate that large FMs, due to their extensive parameter sizes and pre-training on general tasks, achieve significantly lower training loss in one-shot federated fine-tuning compared to smaller models. Our extensive experiments show that one-shot federated fine-tuning not only reduces communication costs but also enables asynchronous aggregation, enhances privacy, and maintains performance consistency with multi-round federated fine-tuning for models larger than 1 billion parameters, on text generation and text-to-image generation tasks. Our findings have the potential to revolutionize federated fine-tuning in practice, enhancing efficiency, reducing costs, and expanding accessibility for large-scale models. This breakthrough paves the way for broader adoption and application of federated fine-tuning across various domains.

FedAWS: A Federated Tuning Approach with Adaptive Weight Shrinking for Pre-trained Foundation Models

FedPFT: Federated Proxy Fine-Tuning of Foundation Models

Enhancing Data Quality in Federated Fine-Tuning of Foundation Models

Exploring Parameter-Efficient Fine-Tuning to Enable Foundation Models in Federated Learning

FedBaF: Federated Learning Aggregation Biased by a Foundation Model

FedTune: A Deep Dive into Efficient Federated Fine-Tuning with Pre-trained Transformers

FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learning

FedRA: A Random Allocation Strategy for Federated Tuning to Unleash the Power of Heterogeneous Clients

One Communication Round is All It Needs for Federated Fine-Tuning Foundation Models

Beyond Fine-Tuning: Efficient and Effective Fed-Tuning for Mobile/Web Users

FedPEAT: Convergence of Federated Learning, Parameter-Efficient Fine Tuning, and Emulator Assisted Tuning for Artificial Intelligence Foundation Models with Mobile Edge Computing

Enabling Foundation Models: A Distributed Collaboration Framework Based on Graph Federated Learning

Efficient Federated Prompt Tuning for Black-box Large Pre-trained Models

Federated Fine-Tuning for Pre-Trained Foundation Models Over Wireless Networks

Towards Efficient Model-Heterogeneity Federated Learning for Large Models

Federated Foundation Models on Heterogeneous Time Series

Conquering the Communication Constraints to Enable Large Pre-Trained Models in Federated Learning

FedFTHA: A Fine-Tuning and Head Aggregation Method in Federated Learning

FedMS: Federated Learning with Mixture of Sparsely Activated Foundations Models

FedPT: Federated Proxy-Tuning of Large Language Models on Resource-Constrained Edge Devices

Dual-Personalizing Adapter for Federated Foundation Models