Abstract:Dual first-order methods are essential techniques for large-scale constrained convex optimization. However, when recovering the primal solutions, we need $T(\epsilon^{-2})$ iterations to achieve an $\epsilon$-optimal primal solution when we apply an algorithm to the non-strongly convex dual problem with $T(\epsilon^{-1})$ iterations to achieve an $\epsilon$-optimal dual solution, where $T(x)$ can be $x$ or $\sqrt{x}$. In this paper, we prove that the iteration complexity of the primal solutions and dual solutions have the same $O\left(\frac{1}{\sqrt{\epsilon}}\right)$ order of magnitude for the accelerated randomized dual coordinate ascent. When the dual function further satisfies the quadratic functional growth condition, by restarting the algorithm at any period, we establish the linear iteration complexity for both the primal solutions and dual solutions even if the condition number is unknown. When applied to the regularized empirical risk minimization problem, we prove the iteration complexity of $O\left(n\log n+\sqrt{\frac{n}{\epsilon}}\right)$ in both primal space and dual space, where $n$ is the number of samples. Our result takes out the $\left(\log \frac{1}{\epsilon}\right)$ factor compared with the methods based on smoothing/regularization or Catalyst reduction. As far as we know, this is the first time that the optimal $O\left(\sqrt{\frac{n}{\epsilon}}\right)$ iteration complexity in the primal space is established for the dual coordinate ascent based stochastic algorithms. We also establish the accelerated linear complexity for some problems with nonsmooth loss, i.e., the least absolute deviation and SVM.

On the Linear Convergence of the General First Order Primal-Dual Algorithm

Linear Convergence of First- and Zeroth-Order Primal-Dual Algorithms for Distributed Nonconvex Optimization

An Efficient Interior-Point Algorithm with New Non-Monotone Line Search Filter Method for Nonlinear Constrained Programming

Convergence analysis of approximate primal solutions in dual first-order methods

Linear Convergence of Subgradient Algorithm for Convex Feasibility on Riemannian Manifolds

On Convergence Rates of Linearized Proximal Algorithms for Convex Composite Optimization with Applications.

A new primal-dual algorithm for structured convex optimization involving a lipschitzian term

Convergence analysis of a primal-dual optimization-by-continuation algorithm

New convergence results of the golden ratio primal-dual algorithm

On the Geometry and Refined Rate of Primal-Dual Hybrid Gradient for Linear Programming

A New Primal-Dual Algorithm for a Class of Nonlinear Compositional Convex Optimization Problems

Convergence analysis of a primal–dual optimization-by-continuation algorithm

Linear Convergence of Inexact Descent Method and Inexact Proximal Gradient Algorithms for Lower-Order Regularization Problems

Linear convergence of a primal-dual algorithm for distributed interval optimization

Automated Lyapunov Analysis of Primal-Dual Optimization Algorithms: An Interpolation Approach

Linear convergence of first order methods for non-strongly convex optimization

Convergence Analysis of Gradient Algorithms on Riemannian Manifolds Without Curvature Constraints and Application to Riemannian Mass

A fast primal-dual algorithm via dynamical system with variable mass for linearly constrained convex optimization

Variational Analysis Perspective on Linear Convergence of Some First Order Methods for Nonsmooth Convex Optimization Problems

On the Complexity Analysis of the Primal Solutions for the Accelerated Randomized Dual Coordinate Ascent

convergence rates of (dual-primal) balanced augmented Lagrangian methods for linearly constrained convex programming