Abstract:Large Language Models (LLMs) have demonstrated remarkable capabilities across various domains, particularly in task generalization for both text and vision data. While fine-tuning these models can significantly enhance their performance on specific downstream tasks, it often requires high-quality data that cannot be shared due to privacy concerns. Federated Learning (FL) offers a promising solution for collaborative training without direct data sharing. However, many parameter-efficient fine-tuning strategies for LLMs in FL, particularly those based on Low-Rank Adaptation (LoRA), face limitations. In this paper, we critically analyze the convergence and performance guarantees of popular FL frameworks utilizing LoRA, highlighting its suboptimal nature due to constrained subspace learning of low-rank matrices. This limitation hinders effective fine-tuning of LLMs in federated settings. Through rigorous analytical and empirical evaluations, we demonstrate that direct weight averaging outperforms LoRA-based strategies, leading to superior performance for fine-tuned models. Our comprehensive comparison unmasks inefficiencies in LoRA approaches and underscores the advantages of direct weight aggregation. We extend our analysis to low-rank gradient-based optimizers, such as GaLore, used during local training steps. Our findings show that GaLore along with direct-weight aggregation is a more effective approach, outperforming federated LoRA methods like FlexLoRA and FFA-LoRA across both text and image modalities. While privacy remains paramount in FL discourse, our focus is on assessing performance outcomes of federated fine-tuned models and evaluating various FL frameworks from both theoretical and empirical perspectives. Our findings advocate reassessing the reliance on LoRA within FL contexts, paving the way for more efficient training methodologies.

Federated Low-Rank Adaptation for Large Models Fine-Tuning over Wireless Networks

Over-the-air Learning Rate Optimization for Federated Learning

Federated Fine-Tuning for Pre-Trained Foundation Models Over Wireless Networks

FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations

Federated LLMs Fine-tuned with Adaptive Importance-Aware LoRA

Personalized Wireless Federated Learning for Large Language Models

Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Models

Exploring Gradient Subspaces: Addressing and Overcoming LoRA's Limitations in Federated Fine-Tuning of Large Language Models

SA-FedLora: Adaptive Parameter Allocation for Efficient Federated Learning with LoRA Tuning

Towards Robust and Efficient Federated Low-Rank Adaptation with Heterogeneous Clients

Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resources

RBLA: Rank-Based-LoRA-Aggregation for Fine-tuning Heterogeneous Models in FLaaS

Federated Learning in Multi-RIS-Aided Systems

Adaptive Parameter-Efficient Federated Fine-Tuning on Heterogeneous Devices

LoRA-FAIR: Federated LoRA Fine-Tuning with Aggregation and Initialization Refinement

Improving LoRA in Privacy-preserving Federated Learning

FDLoRA: Personalized Federated Learning of Large Language Model via Dual LoRA Tuning

Learning Rate Optimization for Federated Learning Exploiting Over-the-air Computation

Over-the-Air Federated Learning and Optimization

Convergence Analysis and Optimization of Over-the-Air Federated Meta-Learning.

FedsLLM: Federated Split Learning for Large Language Models over Communication Networks