Abstract:Foundation models (FMs) achieve strong performance across diverse tasks with task-specific fine-tuning, yet full parameter fine-tuning is often computationally prohibitive for large models. Parameter-efficient fine-tuning (PEFT) methods like Low-Rank Adaptation (LoRA) reduce this cost by introducing low-rank matrices for tuning fewer parameters. While LoRA allows for efficient fine-tuning, it requires significant data for adaptation, making Federated Learning (FL) an appealing solution due to its privacy-preserving collaborative framework. However, combining LoRA with FL introduces two key challenges: the \textbf{Server-Side LoRA Aggregation Bias}, where server-side averaging of LoRA matrices diverges from the ideal global update, and the \textbf{Client-Side LoRA Initialization Drift}, emphasizing the need for consistent initialization across rounds. Existing approaches address these challenges individually, limiting their effectiveness. We propose LoRA-FAIR, a novel method that tackles both issues by introducing a correction term on the server while keeping the original LoRA modules, enhancing aggregation efficiency and accuracy. LoRA-FAIR maintains computational and communication efficiency, yielding superior performance over state-of-the-art methods. Experimental results on ViT and MLP-Mixer models across large-scale datasets demonstrate that LoRA-FAIR consistently achieves performance improvements in FL settings.

What problem does this paper attempt to address?

### What problems does this paper attempt to solve? This paper aims to solve two major challenges encountered when using Low - Rank Adaptation (LoRA) for Parameter - Efficient Fine - Tuning (PEFT) in the Federated Learning (FL) environment: 1. **Server - Side Aggregation Bias**: - In the traditional FL framework, the server updates the global model by averaging the LoRA modules \( \bar{A} \) and \( \bar{B} \) uploaded by clients. However, directly averaging these low - rank matrices cannot accurately capture the ideal global update \( \Delta W \), thus introducing bias. Specifically, the ideal global update should be \( \Delta W=\sum_{k = 1}^{K}p_kB_kA_k \), while the directly averaged update \( \Delta W'=\bar{B}\bar{A} \) is different from it. 2. **Client - Side Initialization Drift**: - At the beginning of each training round, the client needs to re - initialize the LoRA module. If each initialization is random (such as Gaussian distribution), it will lead to inconsistent initialization among different clients, which in turn affects the convergence and performance of the model. Therefore, maintaining consistent initialization among clients is crucial for stabilizing the training process. To address these two challenges, the authors propose a new method named **LoRA - FAIR**, which optimizes server - side aggregation by introducing correction terms and ensures that clients can obtain consistent and stable initialization at the beginning of each training round. Specifically: - **Server - Side**: LoRA - FAIR introduces a residual update \( \Delta B \), making the aggregated \( \bar{B}'=\bar{B}+\Delta B \) closer to the ideal global update \( \Delta W \). - **Client - Side**: By distributing the corrected global LoRA modules \( \bar{B}' \) and \( \bar{A} \), it ensures the initialization consistency of each training round and reduces the drift phenomenon. Experimental results show that LoRA - FAIR significantly outperforms the existing state - of - the - art methods on multiple large - scale datasets while maintaining computational and communication efficiency. ### Summary This paper solves two key problems when applying LoRA for fine - tuning in the federated learning environment: server - side aggregation bias and client - side initialization drift by proposing the LoRA - FAIR method, thereby improving the performance and stability of the model.

LoRA-FAIR: Federated LoRA Fine-Tuning with Aggregation and Initialization Refinement

Improving LoRA in Privacy-preserving Federated Learning

Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Models

Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resources

SA-FedLora: Adaptive Parameter Allocation for Efficient Federated Learning with LoRA Tuning

FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations

RBLA: Rank-Based-LoRA-Aggregation for Fine-tuning Heterogeneous Models in FLaaS

Robust Federated Finetuning of Foundation Models via Alternating Minimization of LoRA

FairLoRA: Unpacking Bias Mitigation in Vision Models with Fairness-Driven Low-Rank Adaptation

FDLoRA: Personalized Federated Learning of Large Language Model via Dual LoRA Tuning

FeDeRA:Efficient Fine-tuning of Language Models in Federated Learning Leveraging Weight Decomposition

Exact Aggregation for Federated and Efficient Fine-Tuning of Foundation Models

FedLoRA: Model-Heterogeneous Personalized Federated Learning with LoRA Tuning

LoRA-GA: Low-Rank Adaptation with Gradient Approximation

Selective Aggregation for Low-Rank Adaptation in Federated Learning

Fed-piLot: Optimizing LoRA Assignment for Efficient Federated Foundation Model Fine-Tuning

PeriodicLoRA: Breaking the Low-Rank Bottleneck in LoRA Optimization

Federated LoRA with Sparse Communication

LoRA-FA: Memory-efficient Low-rank Adaptation for Large Language Models Fine-tuning

LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Report

Federated LLMs Fine-tuned with Adaptive Importance-Aware LoRA