Abstract:In this paper, we consider the decentralized, stochastic nonconvex strongly-concave (NCSC) minimax problem with nonsmooth regularization terms on both primal and dual variables, wherein a network of $m$ computing agents collaborate via peer-to-peer communications. We consider when the coupling function is in expectation or finite-sum form and the double regularizers are convex functions, applied separately to the primal and dual variables. Our algorithmic framework introduces a Lagrangian multiplier to eliminate the consensus constraint on the dual variable. Coupling this with variance-reduction (VR) techniques, our proposed method, entitled VRLM, by a single neighbor communication per iteration, is able to achieve an $\mathcal{O}(\kappa^3\varepsilon^{-3})$ sample complexity under the general stochastic setting, with either a big-batch or small-batch VR option, where $\kappa$ is the condition number of the problem and $\varepsilon$ is the desired solution accuracy. With a big-batch VR, we can additionally achieve $\mathcal{O}(\kappa^2\varepsilon^{-2})$ communication complexity. Under the special finite-sum setting, our method with a big-batch VR can achieve an $\mathcal{O}(n + \sqrt{n} \kappa^2\varepsilon^{-2})$ sample complexity and $\mathcal{O}(\kappa^2\varepsilon^{-2})$ communication complexity, where $n$ is the number of components in the finite sum. All complexity results match the best-known results achieved by a few existing methods for solving special cases of the problem we consider. To the best of our knowledge, this is the first work which provides convergence guarantees for NCSC minimax problems with general convex nonsmooth regularizers applied to both the primal and dual variables in the decentralized stochastic setting. Numerical experiments are conducted on two machine learning problems. Our code is downloadable from <a class="link-external link-https" href="https://github.com/RPI-OPT/VRLM" rel="external noopener nofollow">this https URL</a>.

Capri: C onsensus A ccelerated P roximal R eweighted I teration for A Class of Nonconvex Minimizations

Linearized Proximal Algorithms with Adaptive Stepsizes for Convex Composite Optimization with Applications

Adaptive Consensus: A network pruning approach for decentralized optimization

An Accelerated Algorithm for Linear Quadratic Optimal Consensus of Heterogeneous Multiagent Systems.

A Unified Contraction Analysis of a Class of Distributed Algorithms for Composite Optimization

Accelerated Alternating Direction Method of Multipliers Gradient Tracking for Distributed Optimization

Distributed Algorithms for Composite Optimization: Unified Framework and Convergence Analysis

Distributed Stochastic Consensus Optimization With Momentum for Nonconvex Nonsmooth Problems

Decentralized projected Riemannian stochastic recursive momentum method for smooth optimization on compact submanifolds

Multi-consensus Decentralized Accelerated Gradient Descent

Decentralized projected Riemannian gradient method for smooth optimization on compact submanifolds

Accelerating Distributed Optimization via Fixed-time Convergent Flows: Extensions to Non-convex Functions and Consistent Discretization

Can Decentralized Stochastic Minimax Optimization Algorithms Converge Linearly for Finite-Sum Nonconvex-Nonconcave Problems?

Variance-reduced accelerated methods for decentralized stochastic double-regularized nonconvex strongly-concave minimax problems

A Communication-Efficient Decentralized Newton's Method with Provably Faster Convergence

An Accelerated Stochastic ADMM for Nonconvex and Nonsmooth Finite-Sum Optimization

Surplus-based accelerated algorithms for distributed optimization over directed networks

Accelerated Stochastic Algorithms for Nonconvex Finite-sum and Multi-block Optimization

Decentralized Gradient Descent Maximization Method for Composite Nonconvex Strongly-Concave Minimax Problems

MUSIC: Accelerated Convergence for Distributed Optimization With Inexact and Exact Methods