Abstract:The nonconvex and nonsmooth finite-sum optimization problem with linear constraint has attracted much attention in the fields of artificial intelligence, computer, and mathematics, due to its wide applications in machine learning and the lack of efficient algorithms with convincing convergence theories. A popular approach to solve it is the stochastic Alternating Direction Method of Multipliers (ADMM), but most stochastic ADMM-type methods focus on convex models. In addition, the variance reduction (VR) and acceleration techniques are useful tools in the development of stochastic methods due to their simplicity and practicability in providing acceleration characteristics of various machine learning models. However, it remains unclear whether accelerated SVRG-ADMM algorithm (ASVRG-ADMM), which extends SVRG-ADMM by incorporating momentum techniques, exhibits a comparable acceleration characteristic or convergence rate in the nonconvex setting. To fill this gap, we consider a general nonconvex nonsmooth optimization problem and study the convergence of ASVRG-ADMM. By utilizing a well-defined potential energy function, we establish its sublinear convergence rate $O(1/T)$, where $T$ denotes the iteration number. Furthermore, under the additional Kurdyka-Lojasiewicz (KL) property which is less stringent than the frequently used conditions for showcasing linear convergence rates, such as strong convexity, we show that the ASVRG-ADMM sequence has a finite length and converges to a stationary solution with a linear convergence rate. Several experiments on solving the graph-guided fused lasso problem and regularized logistic regression problem validate that the proposed ASVRG-ADMM performs better than the state-of-the-art methods.

Generalized Majorization-Minimization for Non-Convex Optimization.

Relaxed Majorization-Minimization for Non-Smooth and Non-Convex Optimization

Nonconvex Optimization via MM Algorithms: Convergence Theory

On the Global Convergence of Majorization Minimization Algorithms for Nonconvex Optimization Problems.

The appeals of quadratic majorization–minimization

Stochastic Variance-Reduced Majorization-Minimization Algorithms

Block majorization-minimization with diminishing radius for constrained nonconvex optimization

An Accelerated Stochastic ADMM for Nonconvex and Nonsmooth Finite-Sum Optimization

Universal Majorization-Minimization Algorithms

Majorized Semi-proximal Alternating Coordinate Method for Nonsmooth Convex-Concave Minimax Optimization

Convergence analysis of stochastic higher-order majorization-minimization algorithms

Incremental Majorization-Minimization Optimization with Application to Large-Scale Machine Learning

Faster Stochastic Alternating Direction Method of Multipliers for Nonconvex Optimization

MM for Penalized Estimation

Surrogate Maximization/minimization Algorithms and Extensions

A Stochastic Alternating Direction Method of Multipliers for Non-smooth and Non-convex Optimization

A Proximal-Proximal Majorization-Minimization Algorithm for Nonconvex Rank Regression Problems.

An Accelerated Variance Reduction Stochastic ADMM for Nonconvex Nonsmooth Optimization

Stochastic Variable Metric Proximal Gradient with variance reduction for non-convex composite optimization

On the Q-linear Convergence of a Majorized Proximal ADMM for Convex Composite Programming and Its Applications to Regularized Logistic Regression