Abstract:Classic Machine Learning techniques require training on data available in a single data lake. However, aggregating data from different owners is not always convenient for different reasons, including security, privacy and secrecy. Data carry a value that might vanish when shared with others; the ability to avoid sharing the data enables industrial applications where security and privacy are of paramount importance, making it possible to train global models by implementing only local policies which can be run independently and even on air-gapped data centres. Federated Learning (FL) is a distributed machine learning approach which has emerged as an effective way to address privacy concerns by only sharing local AI models while keeping the data decentralized. Two critical challenges of Federated Learning are managing the heterogeneous systems in the same federated network and dealing with real data, which are often not independently and identically distributed (non-IID) among the clients. In this paper, we focus on the second problem, i.e., the problem of statistical heterogeneity of the data in the same federated network. In this setting, local models might be strayed far from the local optimum of the complete dataset, thus possibly hindering the convergence of the federated model. Several Federated Learning algorithms, such as FedAvg, FedProx and Federated Curvature (FedCurv), aiming at tackling the non-IID setting, have already been proposed. This work provides an empirical assessment of the behaviour of FedAvg and FedCurv in common non-IID scenarios. Results show that the number of epochs per round is an important hyper-parameter that, when tuned appropriately, can lead to significant performance gains while reducing the communication cost. As a side product of this work, we release the non-IID version of the datasets we used so to facilitate further comparisons from the FL community.

Handling Non-IID Data in Federated Learning: An Experimental Evaluation Towards Unified Metrics

Optimizing Federated Learning on Non-IID Data Using Local Shapley Value.

Non-IID data in Federated Learning: A Survey with Taxonomy, Metrics, Methods, Frameworks and Future Directions

Dataset Distillation-based Hybrid Federated Learning on Non-IID Data

A Survey of Federated Learning on Non-IID Data

Federated learning on non-IID data: A survey

Federated Learning on Non-IID Data Silos: An Experimental Study

Federated Learning with Non-IID Data: A Survey

Communication-efficient federated continual learning for distributed learning system with Non-IID data

A Clustered Federated Learning Method of User Behavior Analysis Based on Non-IID Data

A review on different techniques used to combat the non-IID and heterogeneous nature of data in FL

Benchmarking FedAvg and FedCurv for Image Classification Tasks

Federated Learning on Non-Independent and Identically Distributed Data

Federated Learning for Non-IID Data Via Unified Feature Learning and Optimization Objective Alignment

Stabilizing and Improving Federated Learning with Non-IID Data and Client Dropout

A Novel Server-side Aggregation Strategy for Federated Learning in Non-IID Situations

An Experimental Study of Data Heterogeneity in Federated Learning Methods for Medical Imaging

Evaluation Framework For Large-scale Federated Learning

Client Selection for Federated Learning With Non-IID Data in Mobile Edge Computing

FedILC: Weighted Geometric Mean and Invariant Gradient Covariance for Federated Learning on Non-IID Data

Rethinking the Representation in Federated Unsupervised Learning with Non-IID Data