Abstract:Federated learning (FL) has emerged as a promising distributed learning paradigm that enables a large number of mobile devices to cooperatively train a model without sharing their raw data. The iterative training process of FL incurs considerable computation and communication overhead. The workers participating in FL are usually heterogeneous and the workers with poor capabilities may become the bottleneck of model training. To address the challenges of resource overhead and system heterogeneity, this paper proposes an efficient FL framework, called FedMP, that improves both computation and communication efficiency over heterogeneous workers through adaptive model pruning. We theoretically analyze the impact of pruning ratio on training performance, and employ a Multi-Armed Bandit based online learning algorithm to adaptively determine different pruning ratios for heterogeneous workers, even without any prior knowledge of their capabilities. As a result, each worker in FedMP can train and transmit the sub-model that fits its own capabilities, accelerating the training process without hurting model accuracy. To prevent the diverse structures of pruned models from affecting the training convergence, we further present a new parameter synchronization scheme, called Residual Recovery Synchronous Parallel (R2SP). Besides, our proposed framework can be extended to the peer-to-peer (P2P) setting. Extensive experiments on physical devices demonstrate that FedMP is effective for different heterogeneous scenarios and data distributions, and can provide up to 4.1× speedup compared to the existing FL methods.

Towards Efficient Federated Learning: Layer-Wise Pruning-Quantization Scheme and Coding Design.

FedLP: Layer-wise Pruning Mechanism for Communication-Computation Efficient Federated Learning

FedPSE: Personalized Sparsification with Element-wise Aggregation for Federated Learning

Deep Learning Based Coded Over-the-Air Computation for Personalized Federated Learning

Adaptive Network Pruning for Wireless Federated Learning

Computation and Communication Efficient Federated Learning With Adaptive Model Pruning

Lazily Aggregated Quantized Gradient Innovation for Communication-Efficient Federated Learning.

HQsFL: A Novel Training Strategy for Constructing High-performance and Quantum-safe Federated Learning

Two-layer accumulated quantized compression for communication-efficient federated learning: TLAQC

FedFQ: Federated Learning with Fine-Grained Quantization

FedAQ: Communication-Efficient Federated Edge Learning via Joint Uplink and Downlink Adaptive Quantization

Communication-Efficient Design for Quantized Decentralized Federated Learning

Enhancing Decentralized Federated Learning with Model Pruning and Adaptive Communication

Mixed-Precision Quantization for Federated Learning on Resource-Constrained Heterogeneous Devices

Federated Learning With Lossy Distributed Source Coding: Analysis and Optimization

FedPrune: Personalized and Communication-Efficient Federated Learning on Non-IID Data

Communication Efficient Federated Learning with Adaptive Quantization

Toward Communication-Learning Trade-Off for Federated Learning at the Network Edge.

Federated Split Learning with Model Pruning and Gradient Quantization in Wireless Networks

FedNC: A Secure and Efficient Federated Learning Method with Network Coding