Abstract:Federated Learning (FL) is a novel approach that allows for collaborative machine learning while preserving data privacy by leveraging models trained on decentralized devices. However, FL faces challenges due to non-uniformly distributed (non-iid) data across clients, which impacts model performance and its generalization capabilities. To tackle the non-iid issue, recent efforts have utilized the global model as a teaching mechanism for local models. However, our pilot study shows that their effectiveness is constrained by imbalanced data distribution, which induces biases in local models and leads to a 'local forgetting' phenomenon, where the ability of models to generalize degrades over time, particularly for underrepresented classes. This paper introduces FedDistill, a framework enhancing the knowledge transfer from the global model to local models, focusing on the issue of imbalanced class distribution. Specifically, FedDistill employs group distillation, segmenting classes based on their frequency in local datasets to facilitate a focused distillation process to classes with fewer samples. Additionally, FedDistill dissects the global model into a feature extractor and a classifier. This separation empowers local models with more generalized data representation capabilities and ensures more accurate classification across all classes. FedDistill mitigates the adverse effects of data imbalance, ensuring that local models do not forget underrepresented classes but instead become more adept at recognizing and classifying them accurately. Our comprehensive experiments demonstrate FedDistill's effectiveness, surpassing existing baselines in accuracy and convergence speed across several benchmark datasets.

Aligning Model Outputs for Class Imbalanced Non-Iid Federated Learning

Improving Federated Relational Data Modeling via Basis Alignment and Weight Penalty

Federated Learning for Non-IID Data Via Unified Feature Learning and Optimization Objective Alignment

Local Learning Matters: Rethinking Data Heterogeneity in Federated Learning

Addressing Class Imbalance in Federated Learning

Optimizing Federated Learning on Non-IID Data Using Local Shapley Value.

Adaptive Idle Model Fusion in Hierarchical Federated Learning for Unbalanced Edge Regions

No One Left Behind: Inclusive Federated Learning over Heterogeneous Devices

Addressing Algorithmic Disparity and Performance Inconsistency in Federated Learning

Client Selection for Federated Learning With Non-IID Data in Mobile Edge Computing

Integrating Local Real Data with Global Gradient Prototypes for Classifier Re-Balancing in Federated Long-Tailed Learning

Joint Local Relational Augmentation and Global Nash Equilibrium for Federated Learning with Non-IID Data

A Survey on Class Imbalance in Federated Learning

Tackling Data Heterogeneity in Federated Learning with Class Prototypes

A Survey of Federated Learning on Non-IID Data

Federated learning on non-IID and long-tailed data via dual-decoupling

FedDistill: Global Model Distillation for Local Model De-Biasing in Non-IID Federated Learning

Utilizing Free Clients in Federated Learning for Focused Model Enhancement

Adaptive Client Clustering for Efficient Federated Learning over Non-IID and Imbalanced Data

Advocating for the Silent: Enhancing Federated Generalization for Non-Participating Clients

ShapleyFL: Robust Federated Learning Based on Shapley Value