Abstract:Federated Learning (FL) is a novel approach that allows for collaborative machine learning while preserving data privacy by leveraging models trained on decentralized devices. However, FL faces challenges due to non-uniformly distributed (non-iid) data across clients, which impacts model performance and its generalization capabilities. To tackle the non-iid issue, recent efforts have utilized the global model as a teaching mechanism for local models. However, our pilot study shows that their effectiveness is constrained by imbalanced data distribution, which induces biases in local models and leads to a 'local forgetting' phenomenon, where the ability of models to generalize degrades over time, particularly for underrepresented classes. This paper introduces FedDistill, a framework enhancing the knowledge transfer from the global model to local models, focusing on the issue of imbalanced class distribution. Specifically, FedDistill employs group distillation, segmenting classes based on their frequency in local datasets to facilitate a focused distillation process to classes with fewer samples. Additionally, FedDistill dissects the global model into a feature extractor and a classifier. This separation empowers local models with more generalized data representation capabilities and ensures more accurate classification across all classes. FedDistill mitigates the adverse effects of data imbalance, ensuring that local models do not forget underrepresented classes but instead become more adept at recognizing and classifying them accurately. Our comprehensive experiments demonstrate FedDistill's effectiveness, surpassing existing baselines in accuracy and convergence speed across several benchmark datasets.

Server-Client Collaborative Distillation for Federated Reinforcement Learning

Ensemble Distillation for Robust Model Fusion in Federated Learning

A Fair Federated Learning Framework with Reinforcement Learning.

DFRD: Data-Free Robustness Distillation for Heterogeneous Federated Learning

Federated Virtual Learning on Heterogeneous Data with Local-global Distillation

Federated Distillation: A Survey

Optimizing Hierarchical Federated Learning: A Reinforcement Learning Approach

FedDdrl: Federated Double Deep Reinforcement Learning for Heterogeneous IoT with Adaptive Early Client Termination and Local Epoch Adjustment

Towards Client Driven Federated Learning

Handling Data Heterogeneity in Federated Learning via Knowledge Distillation and Fusion

On-Demand Model and Client Deployment in Federated Learning with Deep Reinforcement Learning

FedDistill: Global Model Distillation for Local Model De-Biasing in Non-IID Federated Learning

A Multi-agent Reinforcement Learning Approach for Efficient Client Selection in Federated Learning

An Optimization Method for Non-IID Federated Learning Based on Deep Reinforcement Learning

Federated Learning via Indirect Server-Client Communications

SR-FDIL: Synergistic Replay for Federated Domain-Incremental Learning

Continual Federated Learning Based on Knowledge Distillation

Joint Local Relational Augmentation and Global Nash Equilibrium for Federated Learning with Non-IID Data

Federated Ensemble Model-Based Reinforcement Learning in Edge Computing

Communication-Efficient Consensus Mechanism for Federated Reinforcement Learning

Federated Split Learning Via Mutual Knowledge Distillation