Abstract:Abstract Advances in Natural Language Processing (NLP) have been significantly driven by the adoption of comprehensive pretrained language models (PLMs) such as BERT, RoBERTa. Nevertheless, increasing concerns regarding data privacy and the enforcement of stringent data protection regulations, such as PDPA and GDPR, have highlighted the limitations of traditional centralized machine learning methods. Federated Learning (FL) emerges as a promising solution that mitigates privacy concerns by training models on client devices and aggregating the parameters on a central server, thus avoiding direct data transfer. Despite its considerable potential, FL?s application in NLP faces numerous challenges. These include the need to replicate architectural designs across different devices, the management of non-Independent and Identically Distributed (non-IID) data, and the significant communication overhead caused by frequent transfers of model parameters. Federated Distillation (FD) has been proposed to overcome these challenges by facilitating information transfer through unlabeled public proxy datasets. This approach helps to reduce communication costs and promotes collaboration among different model architectures. However, FD carries potential privacy risks and may result in substantial loss of previously acquired knowledge, thus diminishing the model's effectiveness. To address these issues, we introduce the Privacy-preserving Federated Distillation Method for Pretraining Language Models (PFDP). Our approach distinguishes itself from conventional methods by injecting noise into a select portion of a predetermined dataset, thereby minimizing its impact on the model's utility. Besides, PFDP utilizes transfer learning to improve the generalization abilities of the global model and reduce the impact of catastrophic forgetting. The extensive assessment across many classification tasks illustrates the efficacy of PFDP in enhancing accuracy while safeguarding privacy.

FedBERT: when Federated Learning Meets Pre-training

FEDBFPT: an Efficient Federated Learning Framework for BERT Further Pre-Training

Federated Split BERT for Heterogeneous Text Classification

FedPT: Federated Proxy-Tuning of Large Language Models on Resource-Constrained Edge Devices

FedYolo: Augmenting Federated Learning with Pretrained Transformers

Communication-Efficient and Tensorized Federated Fine-Tuning of Large Language Models

Conquering the Communication Constraints to Enable Large Pre-Trained Models in Federated Learning

FedID: Federated Interactive Distillation for Large-Scale Pretraining Language Models

FedBPT: Efficient Federated Black-box Prompt Tuning for Large Language Models

Federated Learning from Pre-Trained Models: A Contrastive Learning Approach

Efficient Federated Prompt Tuning for Black-box Large Pre-trained Models

PFDP: Privacy-preserving Federated Distillation Method for Pretraining Language Models

Fed-urlBERT: Client-side Lightweight Federated Transformers for URL Threat Analysis

When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methods

bert2BERT: Towards Reusable Pretrained Language Models

Exploring Parameter-Efficient Fine-Tuning to Enable Foundation Models in Federated Learning

FedTune: A Deep Dive into Efficient Federated Fine-Tuning with Pre-trained Transformers

The Future of Large Language Model Pre-training is Federated

FedPEAT: Convergence of Federated Learning, Parameter-Efficient Fine Tuning, and Emulator Assisted Tuning for Artificial Intelligence Foundation Models with Mobile Edge Computing

Pretraining Federated Text Models for Next Word Prediction