Abstract:Federated Learning (FL) is popular for communication-efficient learning from distributed data. To utilize data at different clients without moving them to the cloud, algorithms such as the Federated Averaging (FedAvg) have adopted a computation then aggregation model, in which multiple local updates are performed using local data before aggregation. These algorithms fail to work when faced with practical challenges, e.g., the local data being non-identically independently distributed. In this paper, we first characterize the behavior of the FedAvg algorithm, and show that without strong and unrealistic assumptions on the problem structure, it can behave erratically. Aiming at designing FL algorithms that are provably fast and require as few assumptions as possible, we propose a new algorithm design strategy from the primal-dual optimization perspective. Our strategy yields algorithms that can deal with non-convex objective functions, achieves the best possible optimization and communication complexity (in a well-defined sense), and accommodates full-batch and mini-batch local computation models. Importantly, the proposed algorithms are <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">communication efficient</i> , in that the communication effort can be reduced when the level of heterogeneity among the local data also reduces. In the extreme case where the local data becomes homogeneous, only <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\mathcal {O}(1)$</tex-math></inline-formula> communication is required among the agents. To the best of our knowledge, this is the first algorithmic framework for FL that achieves all the above properties.

On Principled Local Optimization Methods for Federated Learning

Understanding the Training Dynamics in Federated Deep Learning via Aggregation Weight Optimization

Optimizing Federated Learning on Non-IID Data Using Local Shapley Value.

Preconditioned Federated Learning

Accelerated Federated Learning with Decoupled Adaptive Optimization

FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Data.

Communication-Efficient Zeroth-Order Adaptive Optimization for Federated Learning

A Generalized Look at Federated Learning: Survey and Perspectives

Efficient Federated Learning via Local Adaptive Amended Optimizer with Linear Speedup

Local Learning Matters: Rethinking Data Heterogeneity in Federated Learning

Federated Learning with Unbiased Gradient Aggregation and Controllable Meta Updating

Review of Mathematical Optimization in Federated Learning

Personalized Federated Learning: A Unified Framework and Universal Optimization Techniques

CONTINUAL LOCAL TRAINING FOR BETTER INITIALIZATION OF FEDERATED MODELS

Over-the-Air Federated Learning and Optimization

On the Convergence of Communication-Efficient Local SGD for Federated Learning

FedPD: A Federated Learning Framework With Adaptivity to Non-IID Data

When Decentralized Optimization Meets Federated Learning

Addressing Algorithmic Disparity and Performance Inconsistency in Federated Learning

FedGrad: Optimisation in Decentralised Machine Learning

Federated Learning with Additional Mechanisms on Clients to Reduce Communication Costs