Abstract:Federated Learning (FL) is a novel approach that allows for collaborative machine learning while preserving data privacy by leveraging models trained on decentralized devices. However, FL faces challenges due to non-uniformly distributed (non-iid) data across clients, which impacts model performance and its generalization capabilities. To tackle the non-iid issue, recent efforts have utilized the global model as a teaching mechanism for local models. However, our pilot study shows that their effectiveness is constrained by imbalanced data distribution, which induces biases in local models and leads to a 'local forgetting' phenomenon, where the ability of models to generalize degrades over time, particularly for underrepresented classes. This paper introduces FedDistill, a framework enhancing the knowledge transfer from the global model to local models, focusing on the issue of imbalanced class distribution. Specifically, FedDistill employs group distillation, segmenting classes based on their frequency in local datasets to facilitate a focused distillation process to classes with fewer samples. Additionally, FedDistill dissects the global model into a feature extractor and a classifier. This separation empowers local models with more generalized data representation capabilities and ensures more accurate classification across all classes. FedDistill mitigates the adverse effects of data imbalance, ensuring that local models do not forget underrepresented classes but instead become more adept at recognizing and classifying them accurately. Our comprehensive experiments demonstrate FedDistill's effectiveness, surpassing existing baselines in accuracy and convergence speed across several benchmark datasets.

Data Resampling for Federated Learning with Non-IID Labels

Understanding the Training Dynamics in Federated Deep Learning via Aggregation Weight Optimization

Federated Learning with Label Distribution Skew via Logits Calibration.

Dataset Distillation-based Hybrid Federated Learning on Non-IID Data

Optimizing Federated Learning on Non-IID Data Using Local Shapley Value.

Federated learning on non-IID and globally long-tailed data via meta re-weighting networks

FedSampling: A Better Sampling Strategy for Federated Learning

Stabilizing and Improving Federated Learning with Non-IID Data and Client Dropout

Performance Enhancement in Federated Learning by Reducing Class Imbalance of Non-IID Data

FedDistill: Global Model Distillation for Local Model De-Biasing in Non-IID Federated Learning

Federated Learning for distribution skewed data using sample weights

FedSW: Federated learning with adaptive sample weights

FedDNA: Federated Learning with Decoupled Normalization-Layer Aggregation for Non-IID Data

FLAS: Computation and Communication Efficient Federated Learning via Adaptive Sampling

Communication-efficient federated continual learning for distributed learning system with Non-IID data

Federated Learning with Label-Masking Distillation

Rethinking the Representation in Federated Unsupervised Learning with Non-IID Data

Federated Data Quality Assessment Approach: Robust Learning With Mixed Label Noise

A High-Performance Federated Learning Aggregation Algorithm Based on Learning Rate Adjustment and Client Sampling

Federated Two Stage Decoupling With Adaptive Personalization Layers

Federated Learning on Non-Independent and Identically Distributed Data