Abstract:Federated Learning (FL) is popular for communication-efficient learning from distributed data. To utilize data at different clients without moving them to the cloud, algorithms such as the Federated Averaging (FedAvg) have adopted a computation then aggregation model, in which multiple local updates are performed using local data before aggregation. These algorithms fail to work when faced with practical challenges, e.g., the local data being non-identically independently distributed. In this paper, we first characterize the behavior of the FedAvg algorithm, and show that without strong and unrealistic assumptions on the problem structure, it can behave erratically. Aiming at designing FL algorithms that are provably fast and require as few assumptions as possible, we propose a new algorithm design strategy from the primal-dual optimization perspective. Our strategy yields algorithms that can deal with non-convex objective functions, achieves the best possible optimization and communication complexity (in a well-defined sense), and accommodates full-batch and mini-batch local computation models. Importantly, the proposed algorithms are <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">communication efficient</i> , in that the communication effort can be reduced when the level of heterogeneity among the local data also reduces. In the extreme case where the local data becomes homogeneous, only <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\mathcal {O}(1)$</tex-math></inline-formula> communication is required among the agents. To the best of our knowledge, this is the first algorithmic framework for FL that achieves all the above properties.

Federated Matrix Factorization: Algorithm Design and Application to Data Clustering

Optimizing Federated Learning on Non-IID Data Using Local Shapley Value.

A Clustered Federated Learning Method of User Behavior Analysis Based on Non-IID Data

FedEmb: A Vertical and Hybrid Federated Learning Algorithm using Network And Feature Embedding Aggregation

Clustered Federated Learning in Heterogeneous Environment.

FedSC: A federated learning algorithm based on client-side clustering

FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Data.

FedPD: A Federated Learning Framework With Adaptivity to Non-IID Data

An Efficient Client Clustering Algorithm for Clustered Federated Learning

An Aggregation-Free Federated Learning for Tackling Data Heterogeneity

Fuzzy Clustered Federated Learning under Mixed Data Distributions.

FedSeq: A Hybrid Federated Learning Framework Based on Sequential In-Cluster Training.

FedRC: Tackling Diverse Distribution Shifts Challenge in Federated Learning by Robust Clustering

FedGroup: Efficient Clustered Federated Learning via Decomposed Data-Driven Measure

Federated Learning on Non-Independent and Identically Distributed Data

Dual-Segment Clustering Strategy for Federated Learning in Heterogeneous Environments.

Communication-Efficient and Privacy-Preserving Large-Scale Federated Learning Counteracting Heterogeneity

A federated fuzzy c-means clustering algorithm

Federated learning based on stratified sampling and regularization

Federated Cinn Clustering for Accurate Clustered Federated Learning

Accelerating Federated Learning with Cluster Construction and Hierarchical Aggregation.