When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methods

Zhuo Zhang,Yuanhang Yang,Yong Dai,Lizhen Qu,Zenglin Xu

DOI: https://doi.org/10.48550/arXiv.2212.10025

2023-06-02

Abstract:With increasing privacy concerns on data, recent studies have made significant progress using federated learning (FL) on privacy-sensitive natural language processing (NLP) tasks. Much literature suggests fully fine-tuning pre-trained language models (PLMs) in the FL paradigm can mitigate the data heterogeneity problem and close the performance gap with centralized training. However, large PLMs bring the curse of prohibitive communication overhead and local model adaptation costs for the FL system. To this end, we introduce various parameter-efficient tuning (PETuning) methods into federated learning. Specifically, we provide a holistic empirical study of representative PLMs tuning methods in FL. The experimental results cover the analysis of data heterogeneity levels, data scales, and different FL scenarios. Overall communication overhead can be significantly reduced by locally tuning and globally aggregating lightweight model parameters while maintaining acceptable performance in various FL settings. To facilitate the research of PETuning in FL, we also develop a federated tuning framework FedPETuning, which allows practitioners to exploit different PETuning methods under the FL training paradigm conveniently. The source code is available at \url{<a class="link-external link-https" href="https://github.com/iezhuozhuo/FedETuning/tree/deltaTuning" rel="external noopener nofollow">this https URL</a>}.

Machine Learning,Computation and Language

What problem does this paper attempt to address?

### What problems does this paper attempt to solve?

This paper aims to solve two major challenges faced when fine - tuning pre - trained language models (PLMs) in the federated learning (FL) environment:

1. **Communication overhead**: The number of parameters in large - scale PLMs usually reaches millions or even billions, which leads to a huge communication burden when frequently uploading and downloading model parameters during the federated learning process. Limited communication bandwidth may cause severe latency.

2. **Computation and storage costs**: Local clients (such as mobile devices or small - scale servers) usually have limited computational resources and storage space, and it is difficult to support comprehensive fine - tuning of the entire PLM.

To solve these problems, the paper proposes the **Federated Parameter - Efficient Tuning (FedPETuning)** method. By only updating a small part of the parameters in the PLM or adding additional lightweight parameters, FedPETuning can reduce communication overhead and local resource consumption while maintaining model performance and effectively resist privacy attacks.

Specifically, the main contributions of the paper include:
- Providing the first comprehensive benchmark study on parameter - efficient fine - tuning methods (PETuning) for PLMs in the FL environment, covering privacy attacks, performance comparison, and resource constraint analysis.
- Verifying the effectiveness of FedPETuning in reducing communication overhead and storage costs while maintaining high model performance.
- Exploring the performance of FedPETuning under different data heterogeneities and demonstrating its advantages in resisting gradient inversion attacks.

Through these studies, the paper provides important references and guidance for designing more efficient federated learning algorithms in the future.