Abstract:Federated learning (FL) is a distributed machine learning paradigm under privacy preservation. However, data heterogeneity among clients leads to the shared global model obtained after training, which cannot fit the distribution of each client's dataset, and the performance of the model degrades. To address this problem, we proposed a personalized federated learning method based on clustering and knowledge distillation, called pFedCK. In this algorithm, each client has an interactive model that participates in global training and a personalized model that is only trained locally. Both of the models perform knowledge distillation with each other through the feature representation of the middle layer and the soft prediction of the model. In addition, in order to make an interaction model only obtaining the model information from the client, which has similar data distribution and avoids the interference of other heterogeneous information, the server will cluster the clients according to the similarity of the amount of parameter variation uploaded by different interaction models during every training round. By clustering clients, interaction models with similar data distributions can cooperate with each other to better fit the local dataset distribution. Thereby, the performance of personalized model can be improved by obtaining more valuable information indirectly. Finally, we conduct simulation experiments on three benchmark datasets under different data heterogeneity scenarios. Compared to the single model algorithms, the accuracy of pFedCK improved by an average of 23.4% and 23.8% over FedAvg and FedProx, respectively; compared to typical personalization algorithms, the accuracy of pFedCK improved by an average of 0.8% and 1.3%, and a maximum of 1.0% and 2.9% over FedDistill and FML.

One Teacher is Enough: A Server-Clueless Federated Learning with Knowledge Distillation

FedDGP: Disentangling Global and Personal Models for Federated Learning

Fine-tuning Global Model Via Data-Free Knowledge Distillation for Non-IID Federated Learning

Data-Free Knowledge Distillation for Heterogeneous Federated Learning

A Prototype-Based Knowledge Distillation Framework for Heterogeneous Federated Learning

Decentralized Federated Learning Via Mutual Knowledge Distillation.

DFRD: Data-Free Robustness Distillation for Heterogeneous Federated Learning

A Hierarchical Knowledge Transfer Framework for Heterogeneous Federated Learning.

Towards Data-Independent Knowledge Transfer in Model-Heterogeneous Federated Learning

Handling Data Heterogeneity in Federated Learning via Knowledge Distillation and Fusion

Data-free knowledge distillation via generator-free data generation for Non-IID federated learning

Digital Twin-Assisted Knowledge Distillation Framework for Heterogeneous Federated Learning

FedGKD: Towards Heterogeneous Federated Learning via Global Knowledge Distillation

Selective Knowledge Sharing for Privacy-Preserving Federated Distillation without A Good Teacher

A Personalized Federated Learning Method Based on Clustering and Knowledge Distillation

FEDGKD: Toward Heterogeneous Federated Learning Via Global Knowledge Distillation

Data-Free Knowledge Filtering and Distillation in Federated Learning

Personalized and privacy-enhanced federated learning framework via knowledge distillation

FedTKD: A Trustworthy Heterogeneous Federated Learning Based on Adaptive Knowledge Distillation

FedKD: Communication Efficient Federated Learning Via Knowledge Distillation