Abstract:Federated Learning (FL) is a collaborative training paradigm whereby a global Machine Learning (ML) model is trained using typically private and distributed data sources without disclosing the raw data. The approach paves the way for better privacy guarantees, improved overall system scalability, and sustainability. In this context, Federated Averaging (FedAvg) is a representative FL algorithm adopting a client–server protocol that operates in synchronous rounds, where selected learners contribute to the global model via local model updates, trained using their private data, while a server entity aggregates the local contributions, producing the new-generation global model as a weighted average of the local ones. However, when clients possess (highly) dissimilar data, the FedAvg technique becomes ineffective due to divergence in client models. Consequently, FedAvg-trained models struggle to generalize when presented with unseen data from the global distribution. In this research paper, we conduct a systematic review of state-of-the-art approaches proposed to counteract global model performance degradation in the presence of heterogeneous data. To this end, we compile an original taxonomy, highlighting the main algorithmic approaches and mechanisms behind each identified category. Advancing the current body of knowledge, we empirically evaluate the generalization performance on visual tasks of various methods under moderate and significant levels of data heterogeneity, as common practice within the surveyed literature. In addition, the paper benchmarks the performance of hybrid techniques, resulting as a combination of client- and server-side algorithmic tweaks, by shedding light on some associated performance tradeoffs. While recognizing other relevant issues in FL, such as device heterogeneity and energy consumption, which have a non-negligible impact on the learning process, these well-investigated topics are not the main focus of this article.

Enhancing Generalization Robustness of Federated Learning in Highly Heterogeneous Environments

FedDGP: Disentangling Global and Personal Models for Federated Learning

Optimizing Federated Learning on Non-IID Data Using Local Shapley Value.

FedFed: Feature Distillation Against Data Heterogeneity in Federated Learning

A Generalized Look at Federated Learning: Survey and Perspectives

Advancements in Federated Learning: Models, Methods, and Privacy

Privacy preserving federated learning for full heterogeneity

Enhancing generalization in federated learning with heterogeneous data: A comparative literature review

FedCAP: Robust Federated Learning via Customized Aggregation and Personalization

Completely Heterogeneous Federated Learning

Fine-tuning Global Model Via Data-Free Knowledge Distillation for Non-IID Federated Learning

Enhancing Federated Learning by One-Shot Transferring of Intermediate Features from Clients

LF3PFL: A Practical Privacy-Preserving Federated Learning Algorithm Based on Local Federalization Scheme

FedUP: Bridging Fairness and Efficiency in Cross-Silo Federated Learning

Federated Learning for Non-IID Data Via Unified Feature Learning and Optimization Objective Alignment

Tackling Data Heterogeneity in Federated Learning via Loss Decomposition

Acceleration of Federated Learning with Alleviated Forgetting in Local Training

Lazy Aggregation for Heterogeneous Federated Learning

A New Implementation of Federated Learning for Privacy and Security Enhancement

Advocating for the Silent: Enhancing Federated Generalization for Non-Participating Clients

Federated Loss Exploration for Improved Convergence on Non-IID Data