Abstract:Federated learning (FL) enables distribution of machine learning workloads from the cloud to resource-limited edge devices. Unfortunately, current deep networks remain not only too compute-heavy for inference and training on edge devices, but also too large for communicating updates over bandwidth-constrained networks. In this paper, we develop, implement, and experimentally validate a novel FL framework termed Federated Dynamic Sparse Training (FedDST) by which complex neural networks can be deployed and trained with substantially improved efficiency in both on-device computation and in-network communication. At the core of FedDST is a dynamic process that extracts and trains sparse sub-networks from the target full network. With this scheme, "two birds are killed with one stone:" instead of full models, each client performs efficient training of its own sparse networks, and only sparse networks are transmitted between devices and the cloud. Furthermore, our results reveal that the dynamic sparsity during FL training more flexibly accommodates local heterogeneity in FL agents than the fixed, shared sparse masks. Moreover, dynamic sparsity naturally introduces an "in-time self-ensembling effect" into the training dynamics and improves the FL performance even over dense training. In a realistic and challenging non i.i.d. FL setting, FedDST consistently outperforms competing algorithms in our experiments: for instance, at any fixed upload data cap on non-iid CIFAR-10, it gains an impressive accuracy advantage of 10% over FedAvgM when given the same upload data cap; the accuracy gap remains 3% even when FedAvgM is given 2x the upload data cap, further demonstrating efficacy of FedDST. Code is available at: <a class="link-external link-https" href="https://github.com/bibikar/feddst" rel="external noopener nofollow">this https URL</a>.

Unmasking Efficiency: Learning Salient Sparse Models in Non-IID Federated Learning

FedPSE: Personalized Sparsification with Element-wise Aggregation for Federated Learning

Optimizing Federated Learning on Non-IID Data Using Local Shapley Value.

Efficient Semi-Supervised Federated Learning for Heterogeneous Participants

DSFedCon: Dynamic Sparse Federated Contrastive Learning for Data-Driven Intelligent Systems

FED-SDS: Adaptive Structured Dynamic Sparsity for Federated Learning under Heterogeneous Clients

FLAS: Computation and Communication Efficient Federated Learning via Adaptive Sampling

Efficient Model Compression via Global Sparsification for Over-the-Air Federated Learning.

Towards Efficient Federated Learning Framework Via Selective Aggregation of Models

Achieving Personalized Federated Learning with Sparse Local Models

Federated Learning Using Sparse-Adaptive Model Selection for Embedded Edge Computing

Federated Dynamic Sparse Training: Computing Less, Communicating Less, Yet Learning Better

Equitable-FL: Federated Learning with Sparsity for Resource-Constrained Environment

Sparse Federated Learning with Hierarchical Personalization Models.

Clients Help Clients: Alternating Collaboration for Semi-Supervised Federated Learning

Communication-efficient federated continual learning for distributed learning system with Non-IID data

Robust Communication Strategy for Federated Learning by Incorporating Self-Attention

FedSSA: Semantic Similarity-based Aggregation for Efficient Model-Heterogeneous Personalized Federated Learning

Time-Correlated Sparsification for Communication-Efficient Federated Learning

Lottery Aware Sparsity Hunting: Enabling Federated Learning on Resource-Limited Edge

FedSL: A Communication Efficient Federated Learning with Split Layer Aggregation