Abstract:While supervised federated learning approaches have enjoyed significant success, the domain of unsupervised federated learning remains relatively underexplored. Several federated EM algorithms have gained popularity in practice, however, their theoretical foundations are often lacking. In this paper, we first introduce a federated gradient EM algorithm (FedGrEM) designed for the unsupervised learning of mixture models, which supplements the existing federated EM algorithms by considering task heterogeneity and potential adversarial attacks. We present a comprehensive finite-sample theory that holds for general mixture models, then apply this general theory on specific statistical models to characterize the explicit estimation error of model parameters and mixture proportions. Our theory elucidates when and how FedGrEM outperforms local single-task learning with insights extending to existing federated EM algorithms. This bridges the gap between their practical success and theoretical understanding. Our numerical results validate our theory, and demonstrate FedGrEM's superiority over existing unsupervised federated learning benchmarks.

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve This paper aims to address the theoretical foundation issues in Unsupervised Federated Learning (UFL). Specifically, the paper focuses on how to ensure that the Federated Expectation-Maximization algorithm (FedEM) applied in mixed models can be theoretically supported for its success in practical applications. ### Background and Motivation Although Supervised Federated Learning (SFL) has achieved significant success, the field of Unsupervised Federated Learning remains relatively underexplored. Existing federated EM algorithms, while popular in practice, often lack a solid theoretical foundation. Therefore, this paper attempts to fill this gap by introducing a new Federated Gradient EM algorithm (FedGrEM) to address the following key issues: 1. **Task Heterogeneity**: The data distribution across different tasks may vary, requiring the algorithm to adapt to this heterogeneity. 2. **Adversarial Attacks**: In federated learning, the data of some tasks may be tampered with by malicious attackers, and the algorithm needs to be robust to handle such situations. 3. **Theoretical Analysis**: Provide a comprehensive finite-sample theory explaining under what circumstances federated EM algorithms can outperform local single-task learning, and these theoretical results can be applied to specific statistical models. ### Specific Contributions 1. **Algorithm Innovation**: Proposes the FedGrEM algorithm, which not only considers task heterogeneity and potential adversarial attacks but also performs well in terms of computational and communication efficiency. 2. **Theoretical Analysis**: Provides a non-asymptotic theoretical analysis of FedGrEM on general mixed models, detailing the five main components of estimation error: - Iteration error: Gradually vanishes as the number of iterations increases. - Aggregation rate: Depends on the total sample size of non-anomalous tasks. - Heterogeneity cost of mixing proportions. - Task heterogeneity cost. - Cost of anomalous tasks. 3. **Specific Applications**: Applies the theory to Gaussian Mixture Models (GMMs) and Mixture of Regressions (MoRs), demonstrating the superior performance of FedGrEM in these models. 4. **Label Permutation Problem**: Addresses the often-overlooked clustering label permutation problem in federated EM algorithms, ensuring that all non-anomalous tasks share the same permutation in the M-step. ### Conclusion Through these contributions, the paper not only provides a solid theoretical foundation for Unsupervised Federated Learning but also points the way for future research and development. In particular, when tasks have sufficient similarity and the proportion of anomalous tasks is small, FedGrEM can significantly outperform traditional single-task learning methods.

Towards the Theory of Unsupervised Federated Learning: Non-asymptotic Analysis of Federated EM Algorithms

FedPSE: Personalized Sparsification with Element-wise Aggregation for Federated Learning

Network EM Algorithm for Gaussian Mixture Model in Decentralized Federated Learning

FedEmb: A Vertical and Hybrid Federated Learning Algorithm using Network And Feature Embedding Aggregation

AsyncFedED: Asynchronous Federated Learning with Euclidean Distance Based Adaptive Weight Aggregation

Federated Expectation Maximization with heterogeneity mitigation and variance reduction

Adaptive Federated Learning via New Entropy Approach

Emerging Trends in Federated Learning: From Model Fusion to Federated X Learning

Effective Federated Adaptive Gradient Methods with Non-IID Decentralized Data

Compositional federated learning: Applications in distributionally robust averaging and meta learning

Resource-constrained Federated Learning with Heterogeneous Data: Formulation and Analysis

Federated Learning Algorithm Based on Adaptive Gradient Fusion

FedGiA: An Efficient Hybrid Algorithm for Federated Learning

Rethinking the Representation in Federated Unsupervised Learning with Non-IID Data

Achieving Statistical Optimality of Federated Learning: Beyond Stationary Points

Resource-constrained Federated Edge Learning with Heterogeneous Data: Formulation and Analysis

On the Convergence of a Federated Expectation-Maximization Algorithm

Fairness and Accuracy in Federated Learning

FedUC: A Unified Clustering Approach for Hierarchical Federated Learning

Data Augmentation Based Federated Learning