Abstract:Federated Learning (FL) is a collaborative training paradigm whereby a global Machine Learning (ML) model is trained using typically private and distributed data sources without disclosing the raw data. The approach paves the way for better privacy guarantees, improved overall system scalability, and sustainability. In this context, Federated Averaging (FedAvg) is a representative FL algorithm adopting a client–server protocol that operates in synchronous rounds, where selected learners contribute to the global model via local model updates, trained using their private data, while a server entity aggregates the local contributions, producing the new-generation global model as a weighted average of the local ones. However, when clients possess (highly) dissimilar data, the FedAvg technique becomes ineffective due to divergence in client models. Consequently, FedAvg-trained models struggle to generalize when presented with unseen data from the global distribution. In this research paper, we conduct a systematic review of state-of-the-art approaches proposed to counteract global model performance degradation in the presence of heterogeneous data. To this end, we compile an original taxonomy, highlighting the main algorithmic approaches and mechanisms behind each identified category. Advancing the current body of knowledge, we empirically evaluate the generalization performance on visual tasks of various methods under moderate and significant levels of data heterogeneity, as common practice within the surveyed literature. In addition, the paper benchmarks the performance of hybrid techniques, resulting as a combination of client- and server-side algorithmic tweaks, by shedding light on some associated performance tradeoffs. While recognizing other relevant issues in FL, such as device heterogeneity and energy consumption, which have a non-negligible impact on the learning process, these well-investigated topics are not the main focus of this article.

FedFN: Feature Normalization for Alleviating Data Heterogeneity Problem in Federated Learning

Feature Norm Regularized Federated Learning: Utilizing Data Disparities for Model Performance Gains

FedBN: Federated Learning on Non-IID Features via Local Batch Normalization

Feature Norm Regularized Federated Learning: Transforming Skewed Distributions into Global Insights

FedWon: Triumphing Multi-domain Federated Learning Without Normalization

FedNNNN: Norm-Normalized Neural Network Aggregation for Fast and Accurate Federated Learning

FedFed: Feature Distillation Against Data Heterogeneity in Federated Learning

Addressing Data Heterogeneity in Federated Learning with Adaptive Normalization-Free Feature Recalibration

FedNor: A Robust Training Framework for Federated Learning Based on Normal Aggregation

A Simple Data Augmentation for Feature Distribution Skewed Federated Learning

Enhancing generalization in federated learning with heterogeneous data: A comparative literature review

Mitigating Data Heterogeneity in Federated Learning with Data Augmentation

Client Contribution Normalization for Enhanced Federated Learning

Fractal Augmented Pre-training and Gaussian Virtual Feature Calibration for Tackling Data Heterogeneity in Federated Learning

Improving Generalization and Personalization in Model-Heterogeneous Federated Learning

An Aggregation-Free Federated Learning for Tackling Data Heterogeneity

FedNorm: an Efficient Federated Learning Framework with Dual Heterogeneity Coexistence on Edge Intelligence Systems.

Why Batch Normalization Damage Federated Learning on Non-IID Data?

Enhancing Generalization Robustness of Federated Learning in Highly Heterogeneous Environments

Gradient Masked Averaging for Federated Learning

Experimenting With Normalization Layers in Federated Learning on Non-IID Scenarios