Abstract:Federated learning (FL) enables distribution of machine learning workloads from the cloud to resource-limited edge devices. Unfortunately, current deep networks remain not only too compute-heavy for inference and training on edge devices, but also too large for communicating updates over bandwidth-constrained networks. In this paper, we develop, implement, and experimentally validate a novel FL framework termed Federated Dynamic Sparse Training (FedDST) by which complex neural networks can be deployed and trained with substantially improved efficiency in both on-device computation and in-network communication. At the core of FedDST is a dynamic process that extracts and trains sparse sub-networks from the target full network. With this scheme, "two birds are killed with one stone:" instead of full models, each client performs efficient training of its own sparse networks, and only sparse networks are transmitted between devices and the cloud. Furthermore, our results reveal that the dynamic sparsity during FL training more flexibly accommodates local heterogeneity in FL agents than the fixed, shared sparse masks. Moreover, dynamic sparsity naturally introduces an "in-time self-ensembling effect" into the training dynamics and improves the FL performance even over dense training. In a realistic and challenging non i.i.d. FL setting, FedDST consistently outperforms competing algorithms in our experiments: for instance, at any fixed upload data cap on non-iid CIFAR-10, it gains an impressive accuracy advantage of 10% over FedAvgM when given the same upload data cap; the accuracy gap remains 3% even when FedAvgM is given 2x the upload data cap, further demonstrating efficacy of FedDST. Code is available at: <a class="link-external link-https" href="https://github.com/bibikar/feddst" rel="external noopener nofollow">this https URL</a>.

Digital Twin-driven Federated Learning for Converged Computing and Networking at the Edge

Edge-cloud Collaborative Learning with Federated and Centralized Features

Communication-Efficient Federated Learning for Digital Twin Edge Networks in Industrial IoT

Low-Latency Federated Learning and Blockchain for Edge Association in Digital Twin Empowered 6G Networks

Accelerating Federated Learning with Data and Model Parallelism in Edge Computing

Accelerating Federated Learning with Model Segmentation for Edge Networks

Adaptive Training and Aggregation for Federated Learning in Multi-tier Computing Networks

Federated Learning as a Service for Hierarchical Edge Networks with Heterogeneous Models

Federated Dynamic Sparse Training: Computing Less, Communicating Less, Yet Learning Better

Accelerating Decentralized Federated Learning in Heterogeneous Edge Computing

Enhancing Federated Learning Convergence with Dynamic Data Queue and Data Entropy-driven Participant Selection

An Efficient Asynchronous Federated Learning Protocol for Edge Devices

Over-the-Air Federated Learning in Digital Twins Empowered UAV Swarms

Cloud-Edge-Client Collaborative Learning in Digital Twin Empowered Mobile Networks

Optimizing Federated Learning With Deep Reinforcement Learning for Digital Twin Empowered Industrial IoT

Resource Efficient Asynchronous Federated Learning for Digital Twin Empowered IoT Network

Optimizing Efficient Personalized Federated Learning with Hypernetworks at Edge

Digital Twin-Assisted Federated Learning with Blockchain in Multi-tier Computing Systems

Adaptive Configuration for Heterogeneous Participants in Decentralized Federated Learning

Experimental Evaluation and Analysis of Federated Learning in Edge Computing Environments

FedLC: Accelerating Asynchronous Federated Learning in Edge Computing