Abstract:Dual first-order methods are essential techniques for large-scale constrained convex optimization. However, when recovering the primal solutions, we need $T(\epsilon^{-2})$ iterations to achieve an $\epsilon$-optimal primal solution when we apply an algorithm to the non-strongly convex dual problem with $T(\epsilon^{-1})$ iterations to achieve an $\epsilon$-optimal dual solution, where $T(x)$ can be $x$ or $\sqrt{x}$. In this paper, we prove that the iteration complexity of the primal solutions and dual solutions have the same $O\left(\frac{1}{\sqrt{\epsilon}}\right)$ order of magnitude for the accelerated randomized dual coordinate ascent. When the dual function further satisfies the quadratic functional growth condition, by restarting the algorithm at any period, we establish the linear iteration complexity for both the primal solutions and dual solutions even if the condition number is unknown. When applied to the regularized empirical risk minimization problem, we prove the iteration complexity of $O\left(n\log n+\sqrt{\frac{n}{\epsilon}}\right)$ in both primal space and dual space, where $n$ is the number of samples. Our result takes out the $\left(\log \frac{1}{\epsilon}\right)$ factor compared with the methods based on smoothing/regularization or Catalyst reduction. As far as we know, this is the first time that the optimal $O\left(\sqrt{\frac{n}{\epsilon}}\right)$ iteration complexity in the primal space is established for the dual coordinate ascent based stochastic algorithms. We also establish the accelerated linear complexity for some problems with nonsmooth loss, i.e., the least absolute deviation and SVM.

Non-ergodic Convergence Rate of an Inertial Accelerated Primal-Dual Algorithm for Saddle Point Problems

Accelerated Primal-dual Scheme for a Class of Stochastic Nonconvex-concave Saddle Point Problems

Inertial primal-dual dynamics with Hessian-driven damping and Tikhonov regularization for convex-concave bilinear saddle point problems

A Second Order Primal–Dual Dynamical System for a Convex–Concave Bilinear Saddle Point Problem

Accelerated Primal-Dual Gradient Method for Smooth and Convex-Concave Saddle-Point Problems with Bilinear Coupling

Optimal Primal-Dual Methods for a Class of Saddle Point Problems

Robust Accelerated Primal-Dual Methods for Computing Saddle Points

Accelerated Primal-Dual Proximal Gradient Splitting Methods for Convex-Concave Saddle-Point Problems

Accelerated Primal-Dual Algorithms for Distributed Smooth Convex Optimization over Networks

On the Complexity Analysis of the Primal Solutions for the Accelerated Randomized Dual Coordinate Ascent

A New Primal-Dual Algorithm for a Class of Nonlinear Compositional Convex Optimization Problems

Accelerated First-Order Continuous-Time Algorithm for Solving Convex-Concave Bilinear Saddle Point Problem

Dynamical Primal-Dual Nesterov Accelerated Method and Its Application to Network Optimization

A fast primal-dual algorithm via dynamical system with variable mass for linearly constrained convex optimization

Fast convergence rates and trajectory convergence of a Tikhonov regularized inertial primal\mbox{-}dual dynamical system with time scaling and vanishing damping

Inertial primal-dual algorithms for structured convex optimization

Convergence Rates of Inertial Splitting Schemes for Nonconvex Composite Optimization

New convergence results of the golden ratio primal-dual algorithm

Inertial self-adaptive algorithms for solving non-smooth convex optimization problems

A convex combination based primal-dual algorithm with linesearch for general convex-concave saddle point problems