Abstract:Federated learning techniques provide a promising framework for collaboratively training a machine learning model without sharing users' data, and delivering a security solution to guarantee privacy during the model training of IoT devices. Nonetheless, challenges posed by data heterogeneity and communication resource constraints make it difficult to develop an efficient federated learning algorithm in terms of the low order of convergence rate. It could significantly deteriorate the quality of service for critical machine learning tasks, e.g., facial recognition, which requires an edge-ready, low-power, low-latency training algorithm. To address these challenges, a communication-efficient federated learning approach is proposed in this paper where the momentum technique is leveraged to accelerate the convergence rate while largely reducing the communication requirements. First, a federated multi-task learning framework by which the learning tasks are reformulated by the multi-objective optimization problem is introduced to address the data heterogeneity. The multiple gradient descent algorithm is harnessed to find the common gradient descending direction for all participants so that the common features can be learned and no sacrifice on each clients' performance. Second, to reduce communication costs, a local momentum technique with global information is developed to speed up the convergence rate, where the convergence analysis of the proposed method under non-convex case is studied. It is proved that the proposed local momentum can actually achieve the same acceleration as the global momentum, whereas it is more robust than algorithms that solely rely on the acceleration by the global momentum. Third, the generalization of the proposed acceleration approach is investigated which is demonstrated by the accelerated variation of FedAvg. Finally, the performance of the proposed method on the learning model accuracy, convergence rate, and robustness to data heterogeneity, is investigated by empirical experiments on four public datasets, while a real-world IoT platform is constructed to demonstrate the communication efficiency of the proposed method.

Clustered Federated Learning Based on Momentum Gradient Descent for Heterogeneous Data

Accelerating Federated Learning via Momentum Gradient Descent

Fedcm: Federated learning with client-level momentum

Rethinking the initialization of Momentum in Federated Learning with Heterogeneous Data

A Federated Learning Aggregation That Integrates Clustering and Momentum Correction Based on Historical High-Quality Gradients

Flexible Clustered Federated Learning for Client-Level Data Distribution Shift

Momentum Benefits Non-iid Federated Learning Simply and Provably

Gradient Scheduling with Global Momentum for Asynchronous Federated Learning in Edge Environment

FEDERATED STOCHASTIC GRADIENT DESCENT BEGETS SELF-INDUCED MOMENTUM

FedGroup: Efficient Clustered Federated Learning via Decomposed Data-Driven Measure

FedAC: An Adaptive Clustered Federated Learning Framework for Heterogeneous Data

LCFed: An Efficient Clustered Federated Learning Framework for Heterogeneous Data

Efficient Federated Learning Using Layer-Wise Regulation and Momentum Aggregation*

Comparative Evaluation of Clustered Federated Learning Methods

Pfedgf: Enabling Personalized Federated Learning Via Gradient Fusion

Gradient Masked Averaging for Federated Learning

Fuzzy Clustered Federated Learning under Mixed Data Distributions.

HFML: Heterogeneous Hierarchical Federated Mutual Learning on Non-Iid Data

Accelerating Communication-efficient Federated Multi-Task Learning with Personalization and Fairness

Faster On-Device Training Using New Federated Momentum Algorithm