Abstract:Dual first-order methods are essential techniques for large-scale constrained convex optimization. However, when recovering the primal solutions, we need $T(\epsilon^{-2})$ iterations to achieve an $\epsilon$-optimal primal solution when we apply an algorithm to the non-strongly convex dual problem with $T(\epsilon^{-1})$ iterations to achieve an $\epsilon$-optimal dual solution, where $T(x)$ can be $x$ or $\sqrt{x}$. In this paper, we prove that the iteration complexity of the primal solutions and dual solutions have the same $O\left(\frac{1}{\sqrt{\epsilon}}\right)$ order of magnitude for the accelerated randomized dual coordinate ascent. When the dual function further satisfies the quadratic functional growth condition, by restarting the algorithm at any period, we establish the linear iteration complexity for both the primal solutions and dual solutions even if the condition number is unknown. When applied to the regularized empirical risk minimization problem, we prove the iteration complexity of $O\left(n\log n+\sqrt{\frac{n}{\epsilon}}\right)$ in both primal space and dual space, where $n$ is the number of samples. Our result takes out the $\left(\log \frac{1}{\epsilon}\right)$ factor compared with the methods based on smoothing/regularization or Catalyst reduction. As far as we know, this is the first time that the optimal $O\left(\sqrt{\frac{n}{\epsilon}}\right)$ iteration complexity in the primal space is established for the dual coordinate ascent based stochastic algorithms. We also establish the accelerated linear complexity for some problems with nonsmooth loss, i.e., the least absolute deviation and SVM.

Extracting Dual Solutions via Primal Optimizers

A New Primal-Dual Algorithm for a Class of Nonlinear Compositional Convex Optimization Problems

A splitting primal-dual proximity algorithm for solving composite optimization problems

A primal-dual interior-point relaxation method with global and rapidly local convergence for nonlinear programs

Primal Dual Alternating Proximal Gradient Algorithms for Nonsmooth Nonconvex Minimax Problems with Coupled Linear Constraints

Stochastic Primal-Dual Algorithms with Faster Convergence than $O(1/\sqrt{T})$ for Problems without Bilinear Structure

On the Complexity Analysis of the Primal Solutions for the Accelerated Randomized Dual Coordinate Ascent

A Fast Single-Loop Primal-Dual Algorithm for Non-Convex Functional Constrained Optimization

A Whole New Ball Game: A Primal Accelerated Method for Matrix Games and Minimizing the Maximum of Smooth Functions

Accelerated Primal-dual Scheme for a Class of Stochastic Nonconvex-concave Saddle Point Problems

On the Complexity of a Practical Primal-Dual Coordinate Method

Stochastic Primal-Dual Algorithms with Faster Convergence Than O(1/√T) for Problems Without Bilinear Structure

On the Differentiability of the Primal-Dual Interior-Point Method

A Primal-Dual Algorithm for Faster Distributionally Robust Optimization

Convergence analysis of a primal-dual optimization-by-continuation algorithm

Primal–Dual Stability in Local Optimality

A Primal-Dual Approach to Bilevel Optimization with Multiple Inner Minima

Convergence analysis of approximate primal solutions in dual first-order methods

Convergence analysis of a primal–dual optimization-by-continuation algorithm

Primal-Dual Sequential Subspace Optimization for Saddle-point Problems

Primal-Dual Algorithm for Distributed Optimization with Coupled Constraints