Abstract:Personalized federated learning (PFL) has gained increasing attention due to its success in handling the statistical heterogeneity of participants' local data by building distinct local models for participants. However, existing PFL schemes require the identical architecture and size of participants' models, e.g., the same number of layers in convolutional neural networks (CNN). In addition, the growing privacy issues (e.g., local update leakage to the curious server in model aggregation) have not been resolved in PFL. The utilization of identical model architectures among participants reduces the cost of privacy attacks since only one uniform attack method is required to extract private information, exacerbating the privacy threat. This paper proposes a novel privacy-preserving PFL framework that supports heterogeneous model architectures and sizes in delivering personalized models for different participants. Specifically, we utilize participants' knowledge, i.e., the soft predictions of local models on a public dataset, to effectively identify participants with similar data distributions regardless of the specific model architectures used. Based on the participants' knowledge, and their computing and storage capabilities, we employ the affinity propagation (AP) algorithm to implement a multi-level participant clustering mechanism for enabling heterogeneous PFL. Since knowledge is independent of original data, it is considered privacy-preserving during the clustering process. We also devise the ring aggregation algorithm to guarantee participants' privacy during the federated training process. In this way, each participant benefits from other participants with similar data distributions privately and obtains a satisfying personalized model. Furthermore, the cross-cluster knowledge transfer method boosts the personalization performance of weak participants. Sufficient theoretical analyses prove the effectiveness and privacy-preserving capacity of the proposed scheme. Extensive experiments on three benchmark datasets also demonstrate the superiority of our proposed scheme in various settings while maintaining privacy protection, outperforming other state-of-the-art schemes.

pFedKT: Personalized federated learning with dual knowledge transfer

Personalized Federated Learning with Adaptive Feature Aggregation and Knowledge Transfer

FedDGP: Disentangling Global and Personal Models for Federated Learning

Parameterized Knowledge Transfer for Personalized Federated Learning

A Personalized Federated Learning Method Based on Clustering and Knowledge Distillation

The Best of Both Worlds: Accurate Global and Personalized Models through Federated Learning with Data-Free Hyper-Knowledge Distillation

Dual Calibration-based Personalised Federated Learning

FedD2S: Personalized Data-Free Federated Knowledge Distillation

Personalized and privacy-enhanced federated learning framework via knowledge distillation

Privacy-Preserving Heterogeneous Personalized Federated Learning with Knowledge

A Hierarchical Knowledge Transfer Framework for Heterogeneous Federated Learning.

A Prototype-Based Knowledge Distillation Framework for Heterogeneous Federated Learning

PRIOR: Personalized Prior for Reactivating the Information Overlooked in Federated Learning

CDKT-FL: Cross-device knowledge transfer using proxy dataset in federated learning

Multiple Adaptive Prototypes Learning in Personalized Federated Learning

Personalized Edge Intelligence via Federated Self-Knowledge Distillation

DKD-pFed: A novel framework for personalized federated learning via decoupling knowledge distillation and feature decorrelation

FedTweet: Two-fold Knowledge Distillation for non-IID Federated Learning

PGFed: Personalize Each Client's Global Objective for Federated Learning

To Be Global or Personalized: Generalized Federated Learning with Cooperative Adaptation for Data Heterogeneity