Abstract:Federated Learning (FL) is an emerging collaborative machine learning framework where multiple clients train the global model without sharing their own datasets. In FL, the model inconsistency caused by the local data heterogeneity across clients results in the near-orthogonality of client updates, which leads to the global update norm reduction and slows down the convergence. Most previous works focus on eliminating the difference of parameters (or gradients) between the local and global models, which may fail to reflect the model inconsistency due to the complex structure of the machine learning model and the Euclidean space's limitation in meaningful geometric representations. In this paper, we propose FedMRUR by adopting the manifold model fusion scheme and a new global optimizer to alleviate the negative impacts. Concretely, FedMRUR adopts a hyperbolic graph manifold regularizer enforcing the representations of the data in the local and global models are close to each other in a low-dimensional subspace. Because the machine learning model has the graph structure, the distance in hyperbolic space can reflect the model bias better than the Euclidean distance. In this way, FedMRUR exploits the manifold structures of the representations to significantly reduce the model inconsistency. FedMRUR also aggregates the client updates norms as the global update norm, which can appropriately enlarge each client's contribution to the global update, thereby mitigating the norm reduction introduced by the near-orthogonality of client updates. Furthermore, we theoretically prove that our algorithm can achieve a linear speedup property for non-convex setting under partial client <a class="link-external link-http" href="http://participation.Experiments" rel="external noopener nofollow">this http URL</a> demonstrate that FedMRUR can achieve a new state-of-the-art (SOTA) accuracy with less communication.

Efficient Federated Learning Using Layer-Wise Regulation and Momentum Aggregation*

FedDGP: Disentangling Global and Personal Models for Federated Learning

Understanding the Training Dynamics in Federated Deep Learning via Aggregation Weight Optimization

Communication-Efficient Model Aggregation with Layer Divergence Feedback in Federated Learning

Lazy Aggregation for Heterogeneous Federated Learning

Optimizing Federated Learning on Non-IID Data Using Local Shapley Value.

Accelerating Federated Learning via Momentum Gradient Descent

Layer-wise and Dimension-wise Locally Adaptive Federated Learning

Is Aggregation the Only Choice? Federated Learning via Layer-wise Model Recombination

Enhancing Federated Learning Convergence with Dynamic Data Queue and Data Entropy-driven Participant Selection

FedDC: Federated Learning with Non-IID Data via Local Drift Decoupling and Correction

Feature Norm Regularized Federated Learning: Utilizing Data Disparities for Model Performance Gains

FedDW: Distilling Weights through Consistency Optimization in Heterogeneous Federated Learning

Federated Learning with Manifold Regularization and Normalized Update Reaggregation

Adaptive Block-Wise Regularization and Knowledge Distillation for Enhancing Federated Learning

Federated Learning on Non-Independent and Identically Distributed Data

Momentum Benefits Non-iid Federated Learning Simply and Provably

A New Look and Convergence Rate of Federated Multitask Learning With Laplacian Regularization

Efficient Federated Learning Using Dynamic Update and Adaptive Pruning with Momentum on Shared Server Data

Sparse Training for Federated Learning with Regularized Error Correction

Federated Mutual Learning: a Collaborative Machine Learning Method for Heterogeneous Data, Models, and Objectives