Abstract:SIAM Journal on Optimization, Volume 34, Issue 1, Page 1006-1044, March 2024. Minimax problems have recently attracted a lot of research interests. A few efforts have been made to solve decentralized nonconvex strongly-concave (NCSC) minimax-structured optimization; however, all of them focus on smooth problems with at most a constraint on the maximization variable. In this paper, we make the first attempt on solving composite NCSC minimax problems that can have convex nonsmooth terms on both minimization and maximization variables. Our algorithm is designed based on a novel reformulation of the decentralized minimax problem that introduces a multiplier to absorb the dual consensus constraint. The removal of dual consensus constraint enables the most aggressive (i.e., local maximization instead of a gradient ascent step) dual update that leads to the benefit of taking a larger primal stepsize and better complexity results. In addition, the decoupling of the nonsmoothness and consensus on the dual variable eases the analysis of a decentralized algorithm; thus our reformulation creates a new way for interested researchers to design new (and possibly more efficient) decentralized methods on solving NCSC minimax problems. We show a global convergence result of the proposed algorithm and an iteration complexity result to produce a (near) stationary point of the reformulation. Moreover, a relation is established between the (near) stationarities of the reformulation and the original formulation. With this relation, we show that when the dual regularizer is smooth, our algorithm can have lower complexity results (with reduced dependence on a condition number) than existing ones to produce a near-stationary point of the original formulation. Numerical experiments are conducted on a distributionally robust logistic regression to demonstrate the performance of the proposed algorithm.

Convergence Results of a Nested Decentralized Gradient Method for Non-strongly Convex Problems

Convergence in High Probability of Distributed Stochastic Gradient Descent Algorithms

Decentralized Stochastic Subgradient Methods for Nonsmooth Nonconvex Optimization

On Nonconvex Decentralized Gradient Descent

On the Convergence of Decentralized Adaptive Gradient Methods

On the Divergence of Decentralized Non-Convex Optimization

On the convergence analysis of the decentralized projected gradient descent method

Accelerated Primal-Dual Algorithms for Distributed Smooth Convex Optimization over Networks

On the Convergence of Local Descent Methods in Federated Learning

Multi-consensus Decentralized Accelerated Gradient Descent

Decentralized projected Riemannian gradient method for smooth optimization on compact submanifolds

Global Optimality in Distributed Low-rank Matrix Factorization

On the convergence of the distributed proximal point algorithm

Linear Convergence of Primal-Dual Gradient Methods and their Performance in Distributed Optimization

Decentralized Conjugate Gradient and Memoryless BFGS Methods

On Convergence of Distributed Approximate Newton Methods: Globalization, Sharper Bounds and Beyond

Double Averaging and Gradient Projection: Convergence Guarantees for Decentralized Constrained Optimization

Convergence result for the gradient-push algorithm and its application to boost up the Push-DIging algorithm

Convergence Rates Analysis of The Quadratic Penalty Method and Its Applications to Decentralized Distributed Optimization

Decentralized Gradient Descent Maximization Method for Composite Nonconvex Strongly-Concave Minimax Problems

Exponential Convergence of Augmented Primal-dual Gradient Algorithms for Partially Strongly Convex Functions