Abstract:Federated learning (FL) strives to enable collaborative training of deep models on the distributed clients of different data without centrally aggregating raw data and hence improving data privacy. Nevertheless, a central challenge in training classification models in the federated system is learning with non-IID data. Most of the existing work is dedicated to eliminating the heterogeneous influence of non-IID data in a federated system. However, in many real-world FL applications, the co-occurrence of data heterogeneity and long-tailed distribution is unavoidable. The universal class distribution is long-tailed, causing them to become easily biased towards head classes, which severely harms the global model performance. In this work, we also discovered an intriguing fact that the classifier logit vector (i.e., pre-softmax output) introduces a heterogeneity drift during the learning process of local training and global optimization, which harms the convergence as well as model performance. Therefore, motivated by the above finding, we propose a novel logit calibration FL method to solve the joint problem of nonIID and long-tailed data in federated learning, called Federated Learning with Logit Calibration (FedLC). First, we presented a method to mitigate the local update drift by calculating the Wasserstein distance among adjacent client logits and then aggregating similar clients to regulate local training. Second, based on the model ensemble, a new distillation method with logit calibration and class weighting was proposed by exploiting the diversity of local models trained on heterogeneous data, which effectively alleviates the global drift problem under long-tailed distribution. Finally, we evaluated FedLC using a highly non-IID and long-tailed experimental setting, comprehensive experiments on several benchmark datasets demonstrated that FedLC achieved superior performance compared with state-of-the-art FL methods, which fully illustrated the effectiveness of the logit calibration strategy.

Federated Skewed Label Learning with Logits Fusion.

Federated Learning with Label Distribution Skew via Logits Calibration.

Rethinking Client Drift in Federated Learning: A Logit Perspective

Logit Calibration for Non-IID and Long-Tailed Data in Federated Learning.

Fair Federated Learning under Domain Skew with Local Consistency and Domain Diversity

Personalized federated learning based on feature fusion

Exploring Vacant Classes in Label-Skewed Federated Learning

Federated Learning with Label-Masking Distillation

Balanced Federated Semisupervised Learning with Fairness-Aware Pseudo-Labeling.

FedDUAL: A Dual-Strategy with Adaptive Loss and Dynamic Aggregation for Mitigating Data Heterogeneity in Federated Learning

FedSat: A Statistical Aggregation Approach for Class Imbalanced Clients in Federated Learning

Handling Data Heterogeneity in Federated Learning via Knowledge Distillation and Fusion

Aligning Model Outputs for Class Imbalanced Non-Iid Federated Learning

FedDistill: Global Model Distillation for Local Model De-Biasing in Non-IID Federated Learning

FBLG: A Local Graph Based Approach for Handling Dual Skewed Non-IID Data in Federated Learning

Non-IID always Bad? Semi-Supervised Heterogeneous Federated Learning with Local Knowledge Enhancement

CalFAT: Calibrated Federated Adversarial Training with Label Skewness

Stabilizing and Improving Federated Learning with Non-IID Data and Client Dropout

Federated Learning with Extreme Label Skew: A Data Extension Approach

BM-FL: A Balanced Weight Strategy for Multi-stage Federated Learning Against Multi-client Data Skewing

Integrating Local Real Data with Global Gradient Prototypes for Classifier Re-Balancing in Federated Long-Tailed Learning