Abstract:Federated Learning (FL) has emerged as a prominent distributed machine learning framework that enables geographically discrete clients to train a global model collaboratively while preserving their privacy-sensitive data. However, due to the non-independent-and-identically-distributed (Non-IID) data generated by heterogeneous clients, the performances of the conventional federated optimization schemes such as FedAvg and its variants deteriorate, requiring the design to adaptively adjust specific model parameters to alleviate the negative influence of heterogeneity. In this paper, by leveraging entropy as a new metric for assessing the degree of system disorder, we propose an adaptive FEDerated learning algorithm based on ENTropy theory (FedEnt) to alleviate the parameter deviation among heterogeneous clients and achieve fast convergence. Nevertheless, given the data disparity and parameter deviation of heterogeneous clients, determining the optimal dynamic learning rate for each client becomes a challenging task as there is no communication among participating clients during the local training epochs. To enable a decentralized learning rate for each participating client, we first introduce the mean-field terms to estimate the components associated with other clients' local parameters. Furthermore, we provide rigorous theoretical analysis on the existence and determination of the mean-field estimators. Based on the mean-field estimators, the closed-form adaptive learning rate for each client is derived by constructing the Hamilton equation. Moreover, the convergence rate of our proposed FedEnt is proved. The extensive experimental results on the real-world datasets (i.e., MNIST, EMNIST-L, CIFAR10, and CIFAR100) show that our FedEnt algorithm surpasses FedAvg and its variants (i.e., FedAdam, FedProx, and FedDyn) under Non-IID settings and achieves a faster convergence rate.

FedEntropy: Information-entropy-aided Training Optimization of Semi-Supervised Federated Learning

FedDGP: Disentangling Global and Personal Models for Federated Learning

FedPSE: Personalized Sparsification with Element-wise Aggregation for Federated Learning

Adaptive Federated Learning via New Entropy Approach

FedEntropy: Efficient Federated Learning for Non-IID Scenarios Using Maximum Entropy Judgment-based Client Selection

FedEmb: A Vertical and Hybrid Federated Learning Algorithm using Network And Feature Embedding Aggregation

Non-IID always Bad? Semi-Supervised Heterogeneous Federated Learning with Local Knowledge Enhancement

Optimizing Federated Learning on Non-IID Data Using Local Shapley Value.

FedEntropy: Efficient Device Grouping for Federated Learning Using Maximum Entropy Judgment

Semi-Supervised Federated Learning with Non-Iid Data: Algorithm and System Design

Efficient Semi-Supervised Federated Learning for Heterogeneous Participants

Improving Generalization and Personalization in Model-Heterogeneous Federated Learning

Disentangling data distribution for Federated Learning

FedFed: Feature Distillation Against Data Heterogeneity in Federated Learning

Disentangling Client Contributions: Improving Federated Learning Accuracy in the Presence of Heterogeneous Data

FedFC: an Efficient Personalized Federated Learning Method on Non-iid Data

FedEP: Tailoring Attention to Heterogeneous Data Distribution with Entropy Pooling for Decentralized Federated Learning

FedCML: Federated Clustering Mutual Learning with non-IID Data.

FedCE: Personalized Federated Learning Method based on Clustering Ensembles

Federated mutual learning

Fine-tuning Global Model Via Data-Free Knowledge Distillation for Non-IID Federated Learning