Abstract:Originated from distributed learning, federated learning enables privacy-preserved collaboration on a new abstracted level by sharing the model parameters only. While the current research mainly focuses on optimizing learning algorithms and minimizing communication overhead left by distributed learning, there is still a considerable gap when it comes to the real implementation on mobile devices. In this article, we start with an empirical experiment to demonstrate computation heterogeneity is a more pronounced bottleneck than communication on the current generation of battery-powered mobile devices, and the existing methods are haunted by mobile stragglers. Further, non-identically distributed data across the mobile users makes the selection of participants critical to the accuracy and convergence. To tackle the computational and statistical heterogeneity, we utilize data as a tuning knob and propose two efficient polynomial-time algorithms to schedule different workloads on various mobile devices, when data is identically or non-identically distributed. For identically distributed data, we combine partitioning and linear bottleneck assignment to achieve near-optimal training time without accuracy loss. For non-identically distributed data, we convert it into an average cost minimization problem and propose a greedy algorithm to find a reasonable balance between computation time and accuracy. We also establish an offline profiler to quantify the runtime behavior of different devices, which serves as the input to the scheduling algorithms. We conduct extensive experiments on a mobile testbed with two datasets and up to 20 devices. Compared with the common benchmarks, the proposed algorithms achieve 2-100× speedup epoch-wise, 2–7 percent accuracy gain and boost the convergence rate by more than 100 percent on CIFAR10.

On Sample Complexity of Learning Shared Representations: the Asymptotic Regime

Towards Efficient Scheduling of Federated Mobile Devices under Computational and Statistical Heterogeneity

Federated selective aggregation for on-device knowledge amalgamation

Exploiting Shared Representations for Personalized Federated Learning

Distributed Multi-Task Learning with Shared Representation

Distributed Learning of Predictive Structures from Multiple Tasks over Networks

Sharing Knowledge in Multi-Task Deep Reinforcement Learning

Multi-Task Model Personalization for Federated Supervised SVM in Heterogeneous Networks

Differentially Private Federated Learning for Multitask Objective Recognition

Personalized Federated Learning with Feature Alignment and Classifier Collaboration

Adaptive Sharing for Image Classification.

Sample-level Data Selection for Federated Learning.

Efficient secure aggregation for privacy-preserving federated learning based on secret sharing

One for One, or All for All: Equilibria and Optimality of Collaboration in Federated Learning

Device Sampling for Heterogeneous Federated Learning: Theory, Algorithms, and Implementation

Federated Selective Aggregation for Knowledge Amalgamation

Federated Multi-Task Learning under a Mixture of Distributions

Communication-Efficient and Privacy-Preserving Large-Scale Federated Learning Counteracting Heterogeneity

Emerging Trends in Federated Learning: From Model Fusion to Federated X Learning

Few-Shot Model Agnostic Federated Learning

A Survey of What to Share in Federated Learning: Perspectives on Model Utility, Privacy Leakage, and Communication Efficiency