Abstract:Federated learning (FL) enables distribution of machine learning workloads from the cloud to resource-limited edge devices. Unfortunately, current deep networks remain not only too compute-heavy for inference and training on edge devices, but also too large for communicating updates over bandwidth-constrained networks. In this paper, we develop, implement, and experimentally validate a novel FL framework termed Federated Dynamic Sparse Training (FedDST) by which complex neural networks can be deployed and trained with substantially improved efficiency in both on-device computation and in-network communication. At the core of FedDST is a dynamic process that extracts and trains sparse sub-networks from the target full network. With this scheme, "two birds are killed with one stone:" instead of full models, each client performs efficient training of its own sparse networks, and only sparse networks are transmitted between devices and the cloud. Furthermore, our results reveal that the dynamic sparsity during FL training more flexibly accommodates local heterogeneity in FL agents than the fixed, shared sparse masks. Moreover, dynamic sparsity naturally introduces an "in-time self-ensembling effect" into the training dynamics and improves the FL performance even over dense training. In a realistic and challenging non i.i.d. FL setting, FedDST consistently outperforms competing algorithms in our experiments: for instance, at any fixed upload data cap on non-iid CIFAR-10, it gains an impressive accuracy advantage of 10% over FedAvgM when given the same upload data cap; the accuracy gap remains 3% even when FedAvgM is given 2x the upload data cap, further demonstrating efficacy of FedDST. Code is available at: <a class="link-external link-https" href="https://github.com/bibikar/feddst" rel="external noopener nofollow">this https URL</a>.

Federated Dynamical Low-Rank Training with Global Loss Convergence Guarantees

Understanding the Training Dynamics in Federated Deep Learning via Aggregation Weight Optimization

FedDGP: Disentangling Global and Personal Models for Federated Learning

Efficient Federated Learning via Local Adaptive Amended Optimizer with Linear Speedup

Federated Learning Based on Dynamic Regularization

Federated Dynamic Sparse Training: Computing Less, Communicating Less, Yet Learning Better

Enhancing Federated Learning Convergence with Dynamic Data Queue and Data Entropy-driven Participant Selection

Federated Learning: Strategies for Improving Communication Efficiency

Towards Federated Low-Rank Adaptation with Rank-Heterogeneous Communication

DeFed: A Principled Decentralized and Privacy-Preserving Federated Learning Algorithm.

FedLite: A Scalable Approach for Federated Learning on Resource-constrained Clients

Efficient Wireless Federated Learning via Low-Rank Gradient Factorization

Communication-Efficient Federated Low-Rank Update Algorithm and its Connection to Implicit Regularization

Explicit Personalization and Local Training: Double Communication Acceleration in Federated Learning

Gradient Free Personalized Federated Learning

Communication-Efficient Nonconvex Federated Learning with Error Feedback for Uplink and Downlink.

Lazy Aggregation for Heterogeneous Federated Learning

FedHM: Efficient Federated Learning for Heterogeneous Models via Low-rank Factorization

Fusion of Global and Local Knowledge for Personalized Federated Learning

Towards Federated Learning Under Resource Constraints via Layer-wise Training and Depth Dropout

FedLALR: Client-Specific Adaptive Learning Rates Achieve Linear Speedup for Non-IID Data