Abstract:Federated Learning (FL) requires frequent exchange of model parameters, which leads to long communication delay, especially when the network environments of clients vary greatly. Moreover, the parameter server needs to wait for the slowest client (i.e., straggler, which may have the largest model size, lowest computing capability or worst network condition) to upload parameters, which may significantly degrade the communication efficiency. Commonly-used client selection methods such as partial client selection would lead to the waste of computing resources and weaken the generalization of the global model. To tackle this problem, along a different line, in this paper, we advocate the approach of model parameter dropout instead of client selection, and accordingly propose a novel framework of Federated learning scheme with Differential parameter Dropout (FedDD). FedDD consists of two key modules: dropout rate allocation and uploaded parameter selection, which will optimize the model parameter uploading ratios tailored to different clients' heterogeneous conditions and also select the proper set of important model parameters for uploading subject to clients' dropout rate constraints. Specifically, the dropout rate allocation is formulated as a convex optimization problem, taking system heterogeneity, data heterogeneity, and model heterogeneity among clients into consideration. The uploaded parameter selection strategy prioritizes on eliciting important parameters for uploading to speedup convergence. Furthermore, we theoretically analyze the convergence of the proposed FedDD scheme. Extensive performance evaluations demonstrate that the proposed FedDD scheme can achieve outstanding performances in both communication efficiency and model convergence, and also possesses a strong generalization capability to data of rare classes.

GWPF: Communication-efficient federated learning with Gradient-Wise Parameter Freezing

Synchronize Only the Immature Parameters: Communication-Efficient Federated Learning By Freezing Parameters Adaptively

Understanding the Training Dynamics in Federated Deep Learning via Aggregation Weight Optimization

FedDGP: Disentangling Global and Personal Models for Federated Learning

FedPSE: Personalized Sparsification with Element-wise Aggregation for Federated Learning

Adaptive Batchsize Selection and Gradient Compression for Wireless Federated Learning

Conquering the Communication Constraints to Enable Large Pre-Trained Models in Federated Learning

Lazily Aggregated Quantized Gradient Innovation for Communication-Efficient Federated Learning.

Gradient sparsification for efficient wireless federated learning with differential privacy

Training Acceleration Method Based on Parameter Freezing

FedDD: Toward Communication-efficient Federated Learning with Differential Parameter Dropout

Save It All: Enabling Full Parameter Tuning for Federated Large Language Models via Cycle Block Gradient Descent

Adaptive Gradient Sparsification for Efficient Federated Learning: An Online Learning Approach

CG-FedLLM: How to Compress Gradients in Federated Fune-tuning for Large Language Models

Heterogeneity-Aware Memory Efficient Federated Learning via Progressive Layer Freezing

FedBug: A Bottom-Up Gradual Unfreezing Framework for Federated Learning

Communication-Efficient Federated Learning with Acceleration of Global Momentum

Communication-Efficient Federated Learning with Accelerated Client Gradient

Improved Communication Efficiency in Federated Natural Policy Gradient via ADMM-based Gradient Updates

FedAPT: Joint Adaptive Parameter Freezing and Resource Allocation for Communication-Efficient Federated Vehicular Networks