Abstract:Federated learning (FL) is a distributed machine learning paradigm under privacy preservation. However, data heterogeneity among clients leads to the shared global model obtained after training, which cannot fit the distribution of each client's dataset, and the performance of the model degrades. To address this problem, we proposed a personalized federated learning method based on clustering and knowledge distillation, called pFedCK. In this algorithm, each client has an interactive model that participates in global training and a personalized model that is only trained locally. Both of the models perform knowledge distillation with each other through the feature representation of the middle layer and the soft prediction of the model. In addition, in order to make an interaction model only obtaining the model information from the client, which has similar data distribution and avoids the interference of other heterogeneous information, the server will cluster the clients according to the similarity of the amount of parameter variation uploaded by different interaction models during every training round. By clustering clients, interaction models with similar data distributions can cooperate with each other to better fit the local dataset distribution. Thereby, the performance of personalized model can be improved by obtaining more valuable information indirectly. Finally, we conduct simulation experiments on three benchmark datasets under different data heterogeneity scenarios. Compared to the single model algorithms, the accuracy of pFedCK improved by an average of 23.4% and 23.8% over FedAvg and FedProx, respectively; compared to typical personalization algorithms, the accuracy of pFedCK improved by an average of 0.8% and 1.3%, and a maximum of 1.0% and 2.9% over FedDistill and FML.

FedD2S: Personalized Data-Free Federated Knowledge Distillation

FedDGP: Disentangling Global and Personal Models for Federated Learning

The Best of Both Worlds: Accurate Global and Personalized Models through Federated Learning with Data-Free Hyper-Knowledge Distillation

A Personalized Federated Learning Method Based on Clustering and Knowledge Distillation

DKD-pFed: A novel framework for personalized federated learning via decoupling knowledge distillation and feature decorrelation

Fine-tuning Global Model Via Data-Free Knowledge Distillation for Non-IID Federated Learning

pFedKT: Personalized federated learning with dual knowledge transfer

Data-Free Knowledge Distillation for Heterogeneous Federated Learning

Handling Data Heterogeneity in Federated Learning via Knowledge Distillation and Fusion

FedTweet: Two-fold Knowledge Distillation for non-IID Federated Learning

CD^2-pFed: Cyclic Distillation-guided Channel Decoupling for Model Personalization in Federated Learning

Personalized Edge Intelligence via Federated Self-Knowledge Distillation

A Prototype-Based Knowledge Distillation Framework for Heterogeneous Federated Learning

Personalized Federated Learning on Long-Tailed Data Via Knowledge Distillation and Generated Features

Data-free knowledge distillation via generator-free data generation for Non-IID federated learning

Decentralized Federated Learning Via Mutual Knowledge Distillation.

Data-Free Knowledge Filtering and Distillation in Federated Learning

Personalized Federated Learning with Adaptive Feature Aggregation and Knowledge Transfer

pFedKT: Personalized Federated Learning via Knowledge Transfer

Personalized and privacy-enhanced federated learning framework via knowledge distillation

Improving Generalization and Personalization in Model-Heterogeneous Federated Learning