Abstract:Despite their exceptional performance on various tasks after fine-tuning, pre-trained language models (PLMs) face significant challenges due to growing privacy concerns with data in centralized training methods. We consider federated learning (FL) to fine-tune PLMs in this paper. However, the substantial number of parameters in PLMs poses significant difficulties for client devices with limited communication and computational resources. One promising solution is to exploit parameter-efficient fine-tuning (PEFT) into FL, which trains a much smaller set of parameters than full parameter fine-tuning (FFT). Although remarkably improving training efficiency, PEFT methods may lead to degraded performance especially when data across different clients are non i.i.d, as revealed by experimental results. To overcome this, we propose FeDeRA, which extends and improves a widely used PEFT method, i.e., low-rank adaption (LoRA). FeDeRA follows LoRA by decomposing the weight matrices of the PLMs into low-rank matrices, which allows for more efficient computation and parameter updates during fine-tuning. Different from LoRA which simply initializes these low-rank matrices by random sampling or zeros, the proposed FeDeRA initializes these matrices by the results of performing singular value decomposition (SVD) on the pre-trained weight matrices. Extensive experiments across various tasks and datasets show that FeDeRA outperforms the considered PEFT baselines and is comparable to or even surpasses FFT method within the FL setting in terms of task performance. Moreover, FeDeRA requires only 1% trainable paramentes compared to FFT, significantly reducing training time costs by more than 90% to achieve the same task performance level. The experimental results also highlight the robustness of FeDeRA against data heterogeneity, as it maintains stable task performance even as data heterogeneity increases.

Sequential Compression Layers for Efficient Federated Learning in Foundational Models

Efficient Federated Finetuning of Tiny Transformers with Resource-Constrained Devices

Exploring Gradient Subspaces: Addressing and Overcoming LoRA's Limitations in Federated Fine-Tuning of Large Language Models

Fisher Information-based Efficient Curriculum Federated Learning with Large Language Models

Layer-wise and Dimension-wise Locally Adaptive Federated Learning

CG-FedLLM: How to Compress Gradients in Federated Fune-tuning for Large Language Models

Bridging the Gap Between Foundation Models and Heterogeneous Federated Learning

Why Go Full? Elevating Federated Learning Through Partial Network Updates

Exploring Parameter-Efficient Fine-Tuning to Enable Foundation Models in Federated Learning

FedLP: Layer-wise Pruning Mechanism for Communication-Computation Efficient Federated Learning

FedFMSL: Federated Learning of Foundations Models with Sparsely Activated LoRA

FedPFT: Federated Proxy Fine-Tuning of Foundation Models

FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations

eFedLLM: Efficient LLM Inference Based on Federated Learning

Exploring Selective Layer Fine-Tuning in Federated Learning

FLoRA: Enhancing Vision-Language Models with Parameter-Efficient Federated Learning

Improving LoRA in Privacy-preserving Federated Learning

FeDeRA:Efficient Fine-tuning of Language Models in Federated Learning Leveraging Weight Decomposition

Exact Aggregation for Federated and Efficient Fine-Tuning of Foundation Models

Towards Efficient Federated Learning: Layer-Wise Pruning-Quantization Scheme and Coding Design.

Conquering the Communication Constraints to Enable Large Pre-Trained Models in Federated Learning