Abstract:Federated learning is a distributed machine learning framework. It facilitates collaborative modeling among participants without sharing raw user data, providing a feasible solution to address data silos in a secure and privacy-preserving manner. However, data heterogeneity is a major challenge of federated learning as it greatly impacts both the convergence speed during model training and the accuracy of predictions. To address this issue, recent federated learning algorithms have incorporated knowledge distillation as a approach of information sharing. Nevertheless, the majority of current approaches only employ logits averaging to combine participants' knowledge, which can have a detrimental impact on the accuracy of the global model, particularly when certain local models show poor performance. besides, some methods rely on public datasets, thereby compromising the privacy-preserving principle of federated learning. To address the aforementioned concerns, we introduces a new federated learning algorithm named Federated Adaptive Knowledge Distillation (FedAdKD), which utilizes knowledge distillation. After the server completes the aggregation of the global model, FedAdKD dynamically allocates distillation weights based on the individual local model's loss on the public dataset, and performs knowledge distillation on the global model. As the use of public datasets violates the privacy-preserving principle of federated learning, we also proposes the incorporation of generative models to generate data that adheres to the original image distribution. By employing federated learning on the client side to train diffusion models, data is generated that adheres to the original image distribution while maintaining privacy. This generated data is then utilized as the distillation dataset. Experimental findings validate the effectiveness of FedAdKD in addressing the obstacles presented by data heterogeneity. FedAdKD not only mitigates the decline in global model accuracy caused by subpar local models and minimizes knowledge forgetting resulting from direct model aggregation but also improves the generalization capacity of the global model.

DFedSN: Decentralized federated learning based on heterogeneous data in social networks

FedEmb: A Vertical and Hybrid Federated Learning Algorithm using Network And Feature Embedding Aggregation

Communication-Efficient and Privacy-Preserving Large-Scale Federated Learning Counteracting Heterogeneity

FedDKD: Federated Learning with Decentralized Knowledge Distillation

Communication-efficient federated continual learning for distributed learning system with Non-IID data

FedEP: Tailoring Attention to Heterogeneous Data Distribution with Entropy Pooling for Decentralized Federated Learning

DisPFL: Towards Communication-Efficient Personalized Federated Learning via Decentralized Sparse Training

Disentangling data distribution for Federated Learning

Collaboratively Learning Federated Models from Noisy Decentralized Data

Federated Two Stage Decoupling With Adaptive Personalization Layers

Graph Federated Learning Based on the Decentralized Framework

DBFed: Debiasing Federated Learning Framework based on Domain-Independent

Decentralized Federated Learning: A Segmented Gossip Approach

Federated two-stage decoupling with adaptive personalization layers

Fedadkd:heterogeneous federated learning via adaptive knowledge distillation

Tackling Feature Skew in Heterogeneous Federated Learning with Semantic Enhancement

Communication-Efficient Learning of Deep Networks from Decentralized Data

FED-SDS: Adaptive Structured Dynamic Sparsity for Federated Learning under Heterogeneous Clients

Semi-Decentralized Federated Edge Learning for Fast Convergence on Non-IID Data

Privacy preserving federated learning for full heterogeneity

FedHF: A High Fairness Federated Learning Algorithm Based on Deconfliction in Heterogeneous Networks.