Abstract:Large Language Models (LLMs) have demonstrated remarkable capabilities across various domains, particularly in task generalization for both text and vision data. While fine-tuning these models can significantly enhance their performance on specific downstream tasks, it often requires high-quality data that cannot be shared due to privacy concerns. Federated Learning (FL) offers a promising solution for collaborative training without direct data sharing. However, many parameter-efficient fine-tuning strategies for LLMs in FL, particularly those based on Low-Rank Adaptation (LoRA), face limitations. In this paper, we critically analyze the convergence and performance guarantees of popular FL frameworks utilizing LoRA, highlighting its suboptimal nature due to constrained subspace learning of low-rank matrices. This limitation hinders effective fine-tuning of LLMs in federated settings. Through rigorous analytical and empirical evaluations, we demonstrate that direct weight averaging outperforms LoRA-based strategies, leading to superior performance for fine-tuned models. Our comprehensive comparison unmasks inefficiencies in LoRA approaches and underscores the advantages of direct weight aggregation. We extend our analysis to low-rank gradient-based optimizers, such as GaLore, used during local training steps. Our findings show that GaLore along with direct-weight aggregation is a more effective approach, outperforming federated LoRA methods like FlexLoRA and FFA-LoRA across both text and image modalities. While privacy remains paramount in FL discourse, our focus is on assessing performance outcomes of federated fine-tuned models and evaluating various FL frameworks from both theoretical and empirical perspectives. Our findings advocate reassessing the reliance on LoRA within FL contexts, paving the way for more efficient training methodologies.

Exact Aggregation for Federated and Efficient Fine-Tuning of Foundation Models

Selective Aggregation for Low-Rank Adaptation in Federated Learning

FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations

LoRA-FAIR: Federated LoRA Fine-Tuning with Aggregation and Initialization Refinement

Towards Robust and Efficient Federated Low-Rank Adaptation with Heterogeneous Clients

Towards Federated Low-Rank Adaptation with Rank-Heterogeneous Communication

Exploring Gradient Subspaces: Addressing and Overcoming LoRA's Limitations in Federated Fine-Tuning of Large Language Models

Improving LoRA in Privacy-preserving Federated Learning

Robust Federated Finetuning of Foundation Models via Alternating Minimization of LoRA

Federated LLMs Fine-tuned with Adaptive Importance-Aware LoRA

Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Models

Fed-piLot: Optimizing LoRA Assignment for Efficient Federated Foundation Model Fine-Tuning

Batched Low-Rank Adaptation of Foundation Models

RBLA: Rank-Based-LoRA-Aggregation for Fine-tuning Heterogeneous Models in FLaaS

FLoCoRA: Federated learning compression with low-rank adaptation

Adaptive Parameter-Efficient Federated Fine-Tuning on Heterogeneous Devices

Federated LoRA with Sparse Communication

Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resources

Randomized Asymmetric Chain of LoRA: The First Meaningful Theoretical Framework for Low-Rank Adaptation

Riemannian Preconditioned LoRA for Fine-Tuning Foundation Models

SA-FedLora: Adaptive Parameter Allocation for Efficient Federated Learning with LoRA Tuning