Abstract:Federated Learning (FL) is a novel approach that allows for collaborative machine learning while preserving data privacy by leveraging models trained on decentralized devices. However, FL faces challenges due to non-uniformly distributed (non-iid) data across clients, which impacts model performance and its generalization capabilities. To tackle the non-iid issue, recent efforts have utilized the global model as a teaching mechanism for local models. However, our pilot study shows that their effectiveness is constrained by imbalanced data distribution, which induces biases in local models and leads to a 'local forgetting' phenomenon, where the ability of models to generalize degrades over time, particularly for underrepresented classes. This paper introduces FedDistill, a framework enhancing the knowledge transfer from the global model to local models, focusing on the issue of imbalanced class distribution. Specifically, FedDistill employs group distillation, segmenting classes based on their frequency in local datasets to facilitate a focused distillation process to classes with fewer samples. Additionally, FedDistill dissects the global model into a feature extractor and a classifier. This separation empowers local models with more generalized data representation capabilities and ensures more accurate classification across all classes. FedDistill mitigates the adverse effects of data imbalance, ensuring that local models do not forget underrepresented classes but instead become more adept at recognizing and classifying them accurately. Our comprehensive experiments demonstrate FedDistill's effectiveness, surpassing existing baselines in accuracy and convergence speed across several benchmark datasets.

Fine-tuning Global Model Via Data-Free Knowledge Distillation for Non-IID Federated Learning

FedDGP: Disentangling Global and Personal Models for Federated Learning

Understanding the Training Dynamics in Federated Deep Learning via Aggregation Weight Optimization

Data-free knowledge distillation via generator-free data generation for Non-IID federated learning

Data-Free Knowledge Distillation for Heterogeneous Federated Learning

Data-Free Knowledge Filtering and Distillation in Federated Learning

Handling Data Heterogeneity in Federated Learning via Knowledge Distillation and Fusion

FedDistill: Global Model Distillation for Local Model De-Biasing in Non-IID Federated Learning

DFRD: Data-Free Robustness Distillation for Heterogeneous Federated Learning

Local-Global Knowledge Distillation in Heterogeneous Federated Learning with Non-IID Data

One Teacher is Enough: A Server-Clueless Federated Learning with Knowledge Distillation

FedGKD: Towards Heterogeneous Federated Learning via Global Knowledge Distillation

FedTweet: Two-fold Knowledge Distillation for non-IID Federated Learning

Optimizing Federated Learning on Non-IID Data Using Local Shapley Value.

Non-IID Federated Learning Based on Global Knowledge Sharing Without Out-of-Domain Data.

FEDGKD: Toward Heterogeneous Federated Learning Via Global Knowledge Distillation

Decentralized Federated Learning Via Mutual Knowledge Distillation.

Improving Generalization and Personalization in Model-Heterogeneous Federated Learning

Towards Data-Independent Knowledge Transfer in Model-Heterogeneous Federated Learning

FedD2S: Personalized Data-Free Federated Knowledge Distillation

The Best of Both Worlds: Accurate Global and Personalized Models through Federated Learning with Data-Free Hyper-Knowledge Distillation