Abstract:The nonconvex and nonsmooth finite-sum optimization problem with linear constraint has attracted much attention in the fields of artificial intelligence, computer, and mathematics, due to its wide applications in machine learning and the lack of efficient algorithms with convincing convergence theories. A popular approach to solve it is the stochastic Alternating Direction Method of Multipliers (ADMM), but most stochastic ADMM-type methods focus on convex models. In addition, the variance reduction (VR) and acceleration techniques are useful tools in the development of stochastic methods due to their simplicity and practicability in providing acceleration characteristics of various machine learning models. However, it remains unclear whether accelerated SVRG-ADMM algorithm (ASVRG-ADMM), which extends SVRG-ADMM by incorporating momentum techniques, exhibits a comparable acceleration characteristic or convergence rate in the nonconvex setting. To fill this gap, we consider a general nonconvex nonsmooth optimization problem and study the convergence of ASVRG-ADMM. By utilizing a well-defined potential energy function, we establish its sublinear convergence rate O(1/T) , where T denotes the iteration number. Furthermore, under the additional Kurdyka–Lojasiewicz (KL) property which is less stringent than the frequently used conditions for showcasing linear convergence rates, such as strong convexity, we show that the ASVRG-ADMM sequence almost surely has a finite length and converges to a stationary solution with a linear convergence rate. Several experiments on solving the graph-guided fused lasso problem and regularized logistic regression problem validate that the proposed ASVRG-ADMM performs better than the state-of-the-art methods.

Acceleration of the EM Algorithm Using the Vector Aitken Method and Its Steffensen Form

Acceleration of the EM algorithm using the vector epsilon algorithm

Acceleration of the EM Algorithm

Accelerated Stochastic ADMM with Variance Reduction

Damped Anderson acceleration with restarts and monotonicity control for accelerating EM and EM-like algorithms

Stochastic Expectation Maximization with Variance Reduction.

Parameter Expansion to Accelerate Em: the Px-Em Algorithm

Stochastic Momentum Method with Double Acceleration for Regularized Empirical Risk Minimization

A Mixed Evolutionary Algorithm to Solve the O-D Matrix Estimation Problem

The ECME Algorithm: A Simple Extension of EM and ECM with Faster Monotone Convergence

Communication-efficient distributed EM algorithm

Accelerated stochastic admm for empirical risk minimization

String-Averaging Expectation-Maximization for Maximum Likelihood Estimation in Emission Tomography

Asymptotic Convergence Properties of the EM Algorithm with Respect to the Overlap in the Mixture

Deterministic Approximate EM Algorithm; Application to the Riemann Approximation EM and the Tempered EM

Fast convergence of the Expectation Maximization algorithm under a logarithmic Sobolev inequality

Acceleration of the EM and ECM algorithms using the Aitken δ2 method for log-linear models with partially classified data

On the global and componentwise rates of convergence of the EM algorithm

Stochastic Approximation with Biased MCMC for Expectation Maximization

An accelerated stochastic ADMM for nonconvex and nonsmooth finite-sum optimization