Abstract:Model averaging is a widely adopted technique in federated learning (FL) that aggregates multiple client models to obtain a global model. Remarkably, model averaging in FL yields a superior global model, even when client models are trained with non-convex objective functions and on heterogeneous local datasets. However, the rationale behind its success remains poorly understood. To shed light on this issue, we first visualize the loss landscape of FL over client and global models to illustrate their geometric properties. The visualization shows that the client models encompass the global model within a common basin, and interestingly, the global model may deviate from the basin's center while still outperforming the client models. To gain further insights into model averaging in FL, we decompose the expected loss of the global model into five factors related to the client models. Specifically, our analysis reveals that the global model loss after early training mainly arises from \textit{i)} the client model's loss on non-overlapping data between client datasets and the global dataset and \textit{ii)} the maximum distance between the global and client models. Based on the findings from our loss landscape visualization and loss decomposition, we propose utilizing iterative moving averaging (IMA) on the global model at the late training phase to reduce its deviation from the expected minimum, while constraining client exploration to limit the maximum distance between the global and client models. Our experiments demonstrate that incorporating IMA into existing FL methods significantly improves their accuracy and training speed on various heterogeneous data setups of benchmark datasets. Code is available at \url{<a class="link-external link-https" href="https://github.com/TailinZhou/FedIMA" rel="external noopener nofollow">this https URL</a>}.

On the Power of Adaptive Weighted Aggregation in Heterogeneous Federated Learning and Beyond

Understanding the Training Dynamics in Federated Deep Learning via Aggregation Weight Optimization

An Aggregation-Free Federated Learning for Tackling Data Heterogeneity

A Lightweight Method for Tackling Unknown Participation Statistics in Federated Averaging

Communication-Efficient Federated Learning With Data and Client Heterogeneity

AsyncFedED: Asynchronous Federated Learning with Euclidean Distance Based Adaptive Weight Aggregation

Federated Learning under Heterogeneous and Correlated Client Availability

Federated learning with workload-aware client scheduling in heterogeneous systems

Heterogeneity-Aware Federated Learning with Adaptive Client Selection and Gradient Compression.

FedAgg: Adaptive Federated Learning with Aggregated Gradients

FedAA: A Reinforcement Learning Perspective on Adaptive Aggregation for Fair and Robust Federated Learning

Achieving Linear Speedup in Asynchronous Federated Learning with Heterogeneous Clients

Understanding and Improving Model Averaging in Federated Learning on Heterogeneous Data

Federated Learning Optimization Algorithm for Automatic Weight Optimal

Federated Submodel Averaging.

Gradient Masked Averaging for Federated Learning

Federated Learning via Consensus Mechanism on Heterogeneous Data: A New Perspective on Convergence

Revisiting Weighted Aggregation in Federated Learning with Neural Networks

FedPD: A Federated Learning Framework With Adaptivity to Non-IID Data

Federated Learning Aggregation: New Robust Algorithms with Guarantees

Beyond ADMM: A Unified Client-variance-reduced Adaptive Federated Learning Framework