Abstract:In recent years, federated minimax optimization has attracted growing interest due to its extensive applications in various machine learning tasks. While Smoothed Alternative Gradient Descent Ascent (Smoothed-AGDA) has proved its success in centralized nonconvex minimax optimization, how and whether smoothing technique could be helpful in federated setting remains unexplored. In this paper, we propose a new algorithm termed Federated Stochastic Smoothed Gradient Descent Ascent (FESS-GDA), which utilizes the smoothing technique for federated minimax optimization. We prove that FESS-GDA can be uniformly used to solve several classes of federated minimax problems and prove new or better analytical convergence results for these settings. We showcase the practical efficiency of FESS-GDA in practical federated learning tasks of training generative adversarial networks (GANs) and fair classification.

What problem does this paper attempt to address?

This paper discusses the challenges of solving non-convex minimax optimization problems in the context of Federated Learning. Federated Learning allows for collaborative learning without sharing data, thereby protecting data privacy. Traditional Gradient Descent Ascent (GDA) and its stochastic version SGDA are widely used for centralized minimax problems, but their efficiency is not satisfactory in the federated setting. The paper proposes a new algorithm called "Federated Stochastic Smoothed Gradient Descent Ascent" (FESS-GDA), which leverages smoothing techniques to optimize federated minimax problems. Research shows that FESS-GDA can be used to solve various types of federated non-convex minimax problems and provides new or improved convergence results in terms of sample complexity and communication complexity. Specifically, for non-convex-strongly-concave (NC-SC) and non-convex-piecewise-linear (NC-PL) problems, FESS-GDA achieves a sample complexity of O(κ^2m^-1ϵ^-4) while maintaining point stability, and a communication complexity of O(κ^1ϵ^-2), which is significantly better than the previous complexities of O(κ^4m^-1ϵ^-4) and O(κ^2ϵ^-2), respectively. Moreover, FESS-GDA can also handle non-convex-one-dimensional-concave (NC-1PC) and non-convex-concave (NC-C) problems, and in some cases, its performance is comparable to state-of-the-art algorithms with weaker assumptions. The paper also discusses how FESS-GDA converges in different non-convex settings, including specific sample and communication complexity analysis for the PL-PL setting. The authors highlight the practical efficiency of their algorithm in handling certain forms of non-convex-concave problems, such as training Generative Adversarial Networks (GANs) and fair classification tasks. In summary, this paper aims to design a faster and more general minimax optimization algorithm for the federated learning setting by introducing smoothing techniques. It addresses non-convex optimization problems and demonstrates theoretical and experimental advantages in different scenarios.

Stochastic Smoothed Gradient Descent Ascent for Federated Minimax Optimization

A Communication-efficient Algorithm with Linear Convergence for Federated Minimax Learning

Understanding the Training Dynamics in Federated Deep Learning via Aggregation Weight Optimization

Federated Minimax Optimization: Improved Convergence Analyses and Algorithms

Solving a Class of Non-Convex Minimax Optimization in Federated Learning

A Federated Learning Framework for Nonconvex-PL Minimax Problems

FedGiA: An Efficient Hybrid Algorithm for Federated Learning

Fast Decentralized Gradient Tracking for Federated Minimax Optimization with Local Updates

Federated Conditional Stochastic Optimization

Gradient Tracking for High Dimensional Federated Optimization

Accelerated Federated Learning with Decoupled Adaptive Optimization

Federated Optimization with Doubly Regularized Drift Correction

SAGDA: Achieving $\mathcal{O}(ε^{-2})$ Communication Complexity in Federated Min-Max Learning

FADAS: Towards Federated Adaptive Asynchronous Optimization

Faster Adaptive Momentum-Based Federated Methods for Distributed Composition Optimization

FEDERATED STOCHASTIC GRADIENT DESCENT BEGETS SELF-INDUCED MOMENTUM

FedGrad: Optimisation in Decentralised Machine Learning

A Single-Loop Smoothed Gradient Descent-Ascent Algorithm for Nonconvex-Concave Min-Max Problems

Federated Stochastic Gradient Langevin Dynamics

Fairness-aware Federated Minimax Optimization with Convergence Guarantee

Towards Communication-efficient Federated Learning via Sparse and Aligned Adaptive Optimization