Abstract:Pre-trained foundation models (FMs), with extensive number of neurons, are key to advancing next-generation intelligence services, where personalizing these models requires massive amount of task-specific data and computational resources. The prevalent solution involves centralized processing at the edge server, which, however, raises privacy concerns due to the transmission of raw data. Instead, federated fine-tuning (FedFT) is an emerging privacy-preserving fine-tuning (FT) paradigm for personalized pre-trained foundation models. In particular, by integrating low-rank adaptation (LoRA) with federated learning (FL), federated LoRA enables the collaborative FT of a global model with edge devices, achieving comparable learning performance to full FT while training fewer parameters over distributed data and preserving raw data privacy. However, the limited radio resources and computation capabilities of edge devices pose significant challenges for deploying federated LoRA over wireless networks. To this paper, we propose a split federated LoRA framework, which deploys the computationally-intensive encoder of a pre-trained model at the edge server, while keeping the embedding and task modules at the edge devices. Building on this split framework, the paper provides a rigorous analysis of the upper bound of the convergence gap for the wireless federated LoRA system. This analysis motivates the formulation of a long-term upper bound minimization problem, where we decompose the formulated long-term mixed-integer programming (MIP) problem into sequential sub-problems using the Lyapunov technique. We then develop an online algorithm for effective device scheduling and bandwidth allocation. Simulation results demonstrate the effectiveness of the proposed online algorithm in enhancing learning performance.

Inverse Feasibility in Over-the-Air Federated Learning

Boosting Fairness and Robustness in Over-the-Air Federated Learning

Over-The-Air Federated Learning: Status Quo, Open Challenges, and Future Directions

Biased Over-the-Air Federated Learning under Wireless Heterogeneity

Over-the-Air Federated Learning In Broadband Communication

Over-the-Air Federated Learning with Joint Adaptive Computation and Power Control

Enhancement of Over-the-Air Federated Learning by Using AI-based Fluid Antenna System

Over-the-Air Federated Learning and Optimization

RIS-Assisted Over-the-Air Adaptive Federated Learning with Noisy Downlink

Federated Fine-Tuning for Pre-Trained Foundation Models Over Wireless Networks

Toward Secure and Private Over-the-Air Federated Learning

Adaptive Personalized Over-the-Air Federated Learning with Reflecting Intelligent Surfaces

Over-the-Air Computation Empowered Federated Learning: A Joint Uplink-Downlink Design

Convergence Analysis and Optimization of SWIPT-Based Over-the-Air Federated Learning

Differentially Private Federated Learning via Reconfigurable Intelligent Surface

Inferring Linear Feasible Regions using Inverse Optimization

Vertical Federated Learning over Cloud-RAN: Convergence Analysis and System Optimization

ROAR-Fed: RIS-Assisted Over-the-Air Adaptive Resource Allocation for Federated Learning

Balancing Accuracy and Integrity for Reconfigurable Intelligent Surface-aided Over-the-Air Federated Learning

Convergence Analysis and Optimization of Over-the-Air Federated Meta-Learning.

Feasibility-based fixed point networks