Abstract:Fine-tuning large-scale pre-trained models via transfer learning is an emerging important paradigm for a wide range of downstream tasks, with performance heavily reliant on extensive data. Federated learning (FL), as a distributed framework, provides a secure solution to train models on local datasets while safeguarding raw sensitive data. However, FL networks encounter high communication costs due to the massive parameters of large-scale pre-trained models, necessitating parameter-efficient methods. Notably, parameter efficient fine tuning, such as Low-Rank Adaptation (LoRA), has shown remarkable success in fine-tuning pre-trained models. However, prior research indicates that the fixed parameter budget may be prone to the overfitting or slower convergence. To address this challenge, we propose a Simulated Annealing-based Federated Learning with LoRA tuning (SA-FedLoRA) approach by reducing trainable parameters. Specifically, SA-FedLoRA comprises two stages: initiating and annealing. (1) In the initiating stage, we implement a parameter regularization approach during the early rounds of aggregation, aiming to mitigate client drift and accelerate the convergence for the subsequent tuning. (2) In the annealing stage, we allocate higher parameter budget during the early 'heating' phase and then gradually shrink the budget until the 'cooling' phase. This strategy not only facilitates convergence to the global optimum but also reduces communication costs. Experimental results demonstrate that SA-FedLoRA is an efficient FL, achieving superior performance to FedAvg and significantly reducing communication parameters by up to 93.62%.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: in federated learning (FL), the high communication cost problem encountered during the fine - tuning process of large - scale pre - trained models. Specifically: 1. **High communication cost**: Since large - scale pre - trained models contain a large number of parameters, when updating the model in the federated learning framework, these parameters need to be frequently transmitted between the client and the server, resulting in huge communication overhead. 2. **Limitations of fixed parameter budgets**: Although existing parameter - efficient fine - tuning methods (such as LoRA) reduce the number of trainable parameters, the fixed parameter budget may lead to over - fitting or slow convergence problems. To solve these problems, the authors propose the Simulated Annealing - based Federated Learning with LoRA tuning (SA - FedLora) method. By dynamically allocating the parameter budget and adaptively adjusting the rank of LoRA, it accelerates convergence and reduces communication costs while maintaining model performance. ### Key points of the solution - **Two - stage training**: - **Initiating Stage**: Parameter regularization is introduced to align the global and local optimal solutions, alleviate the federated client drift problem, and accelerate the convergence in the subsequent annealing stage. - **Annealing Stage**: A higher parameter budget (high - rank LoRA) is allocated in the early heating stage, and then the parameter budget is gradually reduced (low - rank LoRA) to prevent over - fitting and reduce communication costs. - **Parameter Scheduler**: By defining a parameter scheduler to adaptively adjust the rank of LoRA, including three scheduling strategies: cubic, linear, and cosine, to optimize parameter allocation. ### Experimental results The experimental results show that SA - FedLora significantly reduces communication costs (up to 93.62%) on CIFAR - 10 and the medical face dataset, and performs excellently in terms of accuracy, even outperforming the traditional FedAvg method. ### Summary SA - FedLora effectively solves the high communication cost problem during the fine - tuning of large - scale pre - trained models in federated learning by introducing the simulated annealing mechanism and dynamically adjusting the rank of LoRA, and improves the convergence speed and performance of the model.

SA-FedLora: Adaptive Parameter Allocation for Efficient Federated Learning with LoRA Tuning

LoRA-FAIR: Federated LoRA Fine-Tuning with Aggregation and Initialization Refinement

Improving LoRA in Privacy-preserving Federated Learning

Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resources

FedLoRA: Model-Heterogeneous Personalized Federated Learning with LoRA Tuning

FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations

FDLoRA: Personalized Federated Learning of Large Language Model via Dual LoRA Tuning

FeDeRA:Efficient Fine-tuning of Language Models in Federated Learning Leveraging Weight Decomposition

Federated Fine-Tuning for Pre-Trained Foundation Models Over Wireless Networks

Federated Low-Rank Adaptation for Large Models Fine-Tuning over Wireless Networks

LoRA-SP: Streamlined Partial Parameter Adaptation for Resource-Efficient Fine-Tuning of Large Language Models

Heterogeneous LoRA for Federated Fine-tuning of On-Device Foundation Models

Federated LLMs Fine-tuned with Adaptive Importance-Aware LoRA

Selective Aggregation for Low-Rank Adaptation in Federated Learning

Fed-piLot: Optimizing LoRA Assignment for Efficient Federated Foundation Model Fine-Tuning

Robust Federated Finetuning of Foundation Models via Alternating Minimization of LoRA

IncreLoRA: Incremental Parameter Allocation Method for Parameter-Efficient Fine-tuning

SplitLoRA: A Split Parameter-Efficient Fine-Tuning Framework for Large Language Models

Federated Learning of Large Language Models with Parameter-Efficient Prompt Tuning and Adaptive Optimization

Efficient Federated Finetuning of Tiny Transformers with Resource-Constrained Devices

Federated LoRA with Sparse Communication