Abstract:Federated learning is a distributed machine learning framework. It facilitates collaborative modeling among participants without sharing raw user data, providing a feasible solution to address data silos in a secure and privacy-preserving manner. However, data heterogeneity is a major challenge of federated learning as it greatly impacts both the convergence speed during model training and the accuracy of predictions. To address this issue, recent federated learning algorithms have incorporated knowledge distillation as a approach of information sharing. Nevertheless, the majority of current approaches only employ logits averaging to combine participants' knowledge, which can have a detrimental impact on the accuracy of the global model, particularly when certain local models show poor performance. besides, some methods rely on public datasets, thereby compromising the privacy-preserving principle of federated learning. To address the aforementioned concerns, we introduces a new federated learning algorithm named Federated Adaptive Knowledge Distillation (FedAdKD), which utilizes knowledge distillation. After the server completes the aggregation of the global model, FedAdKD dynamically allocates distillation weights based on the individual local model's loss on the public dataset, and performs knowledge distillation on the global model. As the use of public datasets violates the privacy-preserving principle of federated learning, we also proposes the incorporation of generative models to generate data that adheres to the original image distribution. By employing federated learning on the client side to train diffusion models, data is generated that adheres to the original image distribution while maintaining privacy. This generated data is then utilized as the distillation dataset. Experimental findings validate the effectiveness of FedAdKD in addressing the obstacles presented by data heterogeneity. FedAdKD not only mitigates the decline in global model accuracy caused by subpar local models and minimizes knowledge forgetting resulting from direct model aggregation but also improves the generalization capacity of the global model.

A Multi-Source Credit Data Fusion Approach Based on Federated Distillation Learning

MCKD: Mutually Collaborative Knowledge Distillation for Federated Domain Adaptation and Generalization

A dynamic receptive field and improved feature fusion approach for federated learning in financial credit risk assessment

Federated Learning with Data-Agnostic Distribution Fusion

FedMD: Heterogenous Federated Learning via Model Distillation

Optimization of Personal Credit Evaluation Based on a Federated Deep Learning Model

Ensemble Distillation for Robust Model Fusion in Federated Learning

The Effects of Data Imbalance Under a Federated Learning Approach for Credit Risk Forecasting

Class-Imbalance and Client-Imbalance Federated Learning for Fault Diagnosis

FedDistill: Global Model Distillation for Local Model De-Biasing in Non-IID Federated Learning

Federated Learning Algorithm Based on Adaptive Gradient Fusion

Fedadkd:heterogeneous federated learning via adaptive knowledge distillation

Federated Virtual Learning on Heterogeneous Data with Local-global Distillation

Federated Learning with Label-Masking Distillation

Dataset Distillation-based Hybrid Federated Learning on Non-IID Data

Unlocking the Potential of Federated Learning: The Symphony of Dataset Distillation via Deep Generative Latents

Research on Privacy Protection in Federated Learning Combining Distillation Defense and Blockchain

Handling Data Heterogeneity in Federated Learning via Knowledge Distillation and Fusion

Communication-Efficient Federated Distillation with Active Data Sampling

Privacy-preserving Cost-sensitive Federated Learning from Imbalanced Data

FedDCL: a federated data collaboration learning as a hybrid-type privacy-preserving framework based on federated learning and data collaboration