Abstract:Federated learning is a distributed machine learning framework. It facilitates collaborative modeling among participants without sharing raw user data, providing a feasible solution to address data silos in a secure and privacy-preserving manner. However, data heterogeneity is a major challenge of federated learning as it greatly impacts both the convergence speed during model training and the accuracy of predictions. To address this issue, recent federated learning algorithms have incorporated knowledge distillation as a approach of information sharing. Nevertheless, the majority of current approaches only employ logits averaging to combine participants' knowledge, which can have a detrimental impact on the accuracy of the global model, particularly when certain local models show poor performance. besides, some methods rely on public datasets, thereby compromising the privacy-preserving principle of federated learning. To address the aforementioned concerns, we introduces a new federated learning algorithm named Federated Adaptive Knowledge Distillation (FedAdKD), which utilizes knowledge distillation. After the server completes the aggregation of the global model, FedAdKD dynamically allocates distillation weights based on the individual local model's loss on the public dataset, and performs knowledge distillation on the global model. As the use of public datasets violates the privacy-preserving principle of federated learning, we also proposes the incorporation of generative models to generate data that adheres to the original image distribution. By employing federated learning on the client side to train diffusion models, data is generated that adheres to the original image distribution while maintaining privacy. This generated data is then utilized as the distillation dataset. Experimental findings validate the effectiveness of FedAdKD in addressing the obstacles presented by data heterogeneity. FedAdKD not only mitigates the decline in global model accuracy caused by subpar local models and minimizes knowledge forgetting resulting from direct model aggregation but also improves the generalization capacity of the global model.

FedBiKD: Federated Bidirectional Knowledge Distillation for Distracted Driving Detection

MCKD: Mutually Collaborative Knowledge Distillation for Federated Domain Adaptation and Generalization

When Federated Learning Meets Knowledge Distillation

A Semi-Supervised Federated Learning Scheme Via Knowledge Distillation for Intrusion Detection

Towards driver distraction detection: a privacy-preserving federated learning approach

Enhancing smart road safety with federated learning for Near Crash Detection to advance the development of the Internet of Vehicles

Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutions

Safety Is Our Friend: A Federated Learning Framework Toward Driver’s State and Behavior Detection

Personalized Federated Learning Based on Bidirectional Knowledge Distillation for WiFi Gesture Recognition

Fedadkd:heterogeneous federated learning via adaptive knowledge distillation

Communication-efficient federated learning via knowledge distillation

FedDAF: Federated Deep Attention Fusion for Dangerous Driving Behavior Detection

Personalized and privacy-enhanced federated learning framework via knowledge distillation

Distracted Driving Detection Based on the Fusion of Deep Learning and Causal Reasoning

Digital Twin-Assisted Knowledge Distillation Framework for Heterogeneous Federated Learning

Toward Extremely Lightweight Distracted Driver Recognition With Distillation-Based Neural Architecture Search and Knowledge Transfer

Privacy-Preserved Federated Learning for Autonomous Driving

Edge Bias in Federated Learning and its Solution by Buffered Knowledge Distillation

Federated Deep Learning Meets Autonomous Vehicle Perception: Design and Verification

FedGKD: Towards Heterogeneous Federated Learning via Global Knowledge Distillation

Knowledge Selection and Local Updating Optimization for Federated Knowledge Distillation with Heterogeneous Models.