Abstract:The nonconvex and nonsmooth finite-sum optimization problem with linear constraint has attracted much attention in the fields of artificial intelligence, computer, and mathematics, due to its wide applications in machine learning and the lack of efficient algorithms with convincing convergence theories. A popular approach to solve it is the stochastic Alternating Direction Method of Multipliers (ADMM), but most stochastic ADMM-type methods focus on convex models. In addition, the variance reduction (VR) and acceleration techniques are useful tools in the development of stochastic methods due to their simplicity and practicability in providing acceleration characteristics of various machine learning models. However, it remains unclear whether accelerated SVRG-ADMM algorithm (ASVRG-ADMM), which extends SVRG-ADMM by incorporating momentum techniques, exhibits a comparable acceleration characteristic or convergence rate in the nonconvex setting. To fill this gap, we consider a general nonconvex nonsmooth optimization problem and study the convergence of ASVRG-ADMM. By utilizing a well-defined potential energy function, we establish its sublinear convergence rate $O(1/T)$, where $T$ denotes the iteration number. Furthermore, under the additional Kurdyka-Lojasiewicz (KL) property which is less stringent than the frequently used conditions for showcasing linear convergence rates, such as strong convexity, we show that the ASVRG-ADMM sequence has a finite length and converges to a stationary solution with a linear convergence rate. Several experiments on solving the graph-guided fused lasso problem and regularized logistic regression problem validate that the proposed ASVRG-ADMM performs better than the state-of-the-art methods.

Accelerated Algorithms for Nonconvex Optimization

Accelerated Algorithms for Constrained Convex Optimization

Accelerated Proximal Gradient Methods for Nonconvex Programming

Accelerated gradient methods for nonconvex optimization: Escape trajectories from strict saddle points and convergence to local minima

Accelerated Stochastic Algorithms for Nonconvex Finite-sum and Multi-block Optimization

A Note on the Accelerated Proximal Gradient Method for Nonconvex Optimization

Accelerated gradient methods for nonconvex nonlinear and stochastic programming

Accelerated Stochastic Subgradient Methods under Local Error Bound Condition

Accelerated First-Order Optimization Algorithms for Machine Learning.

More Efficient Accelerated Proximal Algorithm for Nonconvex Problems

Accelerating Asynchronous Algorithms for Convex Optimization by Momentum Compensation

Accelerate Stochastic Subgradient Method by Leveraging Local Growth Condition

Smoothing accelerated algorithm for constrained nonsmooth convex optimization problems

A nonmonotone accelerated proximal gradient method with variable stepsize strategy for nonsmooth and nonconvex minimization problems

Accelerated Parallel Algorithms

Accelerated Quasi-Newton Proximal Extragradient: Faster Rate for Smooth Convex Optimization

Catalyst Acceleration for First-order Convex Optimization: from Theory to Practice

A class of modified accelerated proximal gradient methods for nonsmooth and nonconvex minimization problems

Stagewise Accelerated Stochastic Gradient Methods for Nonconvex Optimization

Convergence Rates of Accelerated Proximal Gradient Algorithms under Independent Noise.

An Accelerated Stochastic ADMM for Nonconvex and Nonsmooth Finite-Sum Optimization