Abstract:Pre-trained foundation models (FMs), with extensive number of neurons, are key to advancing next-generation intelligence services, where personalizing these models requires massive amount of task-specific data and computational resources. The prevalent solution involves centralized processing at the edge server, which, however, raises privacy concerns due to the transmission of raw data. Instead, federated fine-tuning (FedFT) is an emerging privacy-preserving fine-tuning (FT) paradigm for personalized pre-trained foundation models. In particular, by integrating low-rank adaptation (LoRA) with federated learning (FL), federated LoRA enables the collaborative FT of a global model with edge devices, achieving comparable learning performance to full FT while training fewer parameters over distributed data and preserving raw data privacy. However, the limited radio resources and computation capabilities of edge devices pose significant challenges for deploying federated LoRA over wireless networks. To this paper, we propose a split federated LoRA framework, which deploys the computationally-intensive encoder of a pre-trained model at the edge server, while keeping the embedding and task modules at the edge devices. Building on this split framework, the paper provides a rigorous analysis of the upper bound of the convergence gap for the wireless federated LoRA system. This analysis motivates the formulation of a long-term upper bound minimization problem, where we decompose the formulated long-term mixed-integer programming (MIP) problem into sequential sub-problems using the Lyapunov technique. We then develop an online algorithm for effective device scheduling and bandwidth allocation. Simulation results demonstrate the effectiveness of the proposed online algorithm in enhancing learning performance.

FedFMSL: Federated Learning of Foundations Models with Sparsely Activated LoRA

FedMS: Federated Learning with Mixture of Sparsely Activated Foundations Models

Federated Fine-Tuning for Pre-Trained Foundation Models Over Wireless Networks

Federated Generative Learning with Foundation Models

Bridging the Gap Between Foundation Models and Heterogeneous Federated Learning

Federated Foundation Models: Privacy-Preserving and Collaborative Learning for Large Models

A Survey on Efficient Federated Learning Methods for Foundation Model Training

Leveraging Foundation Models for Efficient Federated Learning in Resource-restricted Edge Networks

Synergizing Foundation Models and Federated Learning: A Survey

FedPFT: Federated Proxy Fine-Tuning of Foundation Models

FedBaF: Federated Learning Aggregation Biased by a Foundation Model

Leveraging Foundation Models to Improve Lightweight Clients in Federated Learning

Advances and Open Challenges in Federated Foundation Models

FedsLLM: Federated Split Learning for Large Language Models over Communication Networks

When Foundation Model Meets Federated Learning: Motivations, Challenges, and Future Directions

Fed-piLot: Optimizing LoRA Assignment for Efficient Federated Foundation Model Fine-Tuning

FDLoRA: Personalized Federated Learning of Large Language Model via Dual LoRA Tuning

ZooPFL: Exploring Black-box Foundation Models for Personalized Federated Learning

FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations

CELLM: An Efficient Communication in Large Language Models Training for Federated Learning

Time-FFM: Towards LM-Empowered Federated Foundation Model for Time Series Forecasting