Abstract:Federated Learning (FL) is popular for communication-efficient learning from distributed data. To utilize data at different clients without moving them to the cloud, algorithms such as the Federated Averaging (FedAvg) have adopted a computation then aggregation model, in which multiple local updates are performed using local data before aggregation. These algorithms fail to work when faced with practical challenges, e.g., the local data being non-identically independently distributed. In this paper, we first characterize the behavior of the FedAvg algorithm, and show that without strong and unrealistic assumptions on the problem structure, it can behave erratically. Aiming at designing FL algorithms that are provably fast and require as few assumptions as possible, we propose a new algorithm design strategy from the primal-dual optimization perspective. Our strategy yields algorithms that can deal with non-convex objective functions, achieves the best possible optimization and communication complexity (in a well-defined sense), and accommodates full-batch and mini-batch local computation models. Importantly, the proposed algorithms are <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">communication efficient</i> , in that the communication effort can be reduced when the level of heterogeneity among the local data also reduces. In the extreme case where the local data becomes homogeneous, only <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\mathcal {O}(1)$</tex-math></inline-formula> communication is required among the agents. To the best of our knowledge, this is the first algorithmic framework for FL that achieves all the above properties.

FedAux: an Efficient Framework for Hybrid Federated Learning

FedDGP: Disentangling Global and Personal Models for Federated Learning

FedCross: Towards Accurate Federated Learning via Multi-Model Cross-Aggregation

FedConv: A Learning-on-Model Paradigm for Heterogeneous Federated Clients

Accelerating Hybrid Federated Learning Convergence under Partial Participation

Improving Communication Efficiency: A Dual Adaptive Federated Learning Framework

FedSAE: A Novel Self-Adaptive Federated Learning Framework in Heterogeneous Systems

Federated Learning with Additional Mechanisms on Clients to Reduce Communication Costs

Enhancing Federated Learning by One-Shot Transferring of Intermediate Features from Clients

FedSeq: A Hybrid Federated Learning Framework Based on Sequential In-Cluster Training.

FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Data.

FedGH: Heterogeneous Federated Learning with Generalized Global Header

Improving Generalization and Personalization in Model-Heterogeneous Federated Learning

An Aggregation-Free Federated Learning for Tackling Data Heterogeneity

FedPD: A Federated Learning Framework With Adaptivity to Non-IID Data

AdaptFL: Adaptive Federated Learning Framework for Heterogeneous Devices

Federated Learning with Auxiliary Generator

HybridCom: Improve Federated Learning Efficiency on Unstable Data

Heterogeneity-Aware Federated Learning with Adaptive Client Selection and Gradient Compression.

Adaptive Control of Client Selection and Gradient Compression for Efficient Federated Learning

A Hierarchical Knowledge Transfer Framework for Heterogeneous Federated Learning.