Abstract:In this paper, we consider a class of finite-sum convex optimization problems whose objective function is given by the average of m(≥1)\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$m\, ({\ge }1)$$\end{document} smooth components together with some other relatively simple terms. We first introduce a deterministic primal–dual gradient (PDG) method that can achieve the optimal black-box iteration complexity for solving these composite optimization problems using a primal–dual termination criterion. Our major contribution is to develop a randomized primal–dual gradient (RPDG) method, which needs to compute the gradient of only one randomly selected smooth component at each iteration, but can possibly achieve better complexity than PDG in terms of the total number of gradient evaluations. More specifically, we show that the total number of gradient evaluations performed by RPDG can be O(m)\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$${{\mathcal {O}}} (\sqrt{m})$$\end{document} times smaller, both in expectation and with high probability, than those performed by deterministic optimal first-order methods under favorable situations. We also show that the complexity of the RPDG method is not improvable by developing a new lower complexity bound for a general class of randomized methods for solving large-scale finite-sum convex optimization problems.

Linear Convergence of Random Dual Coordinate Incremental Aggregated Gradient Methods

Linear Convergence of Distributed Aggregative Optimization with Coupled Inequality Constraints

An optimal randomized incremental gradient method

Accelerated Proximal Coordinate Gradient Methods

Convergence Analysis of Gradient Algorithms on Riemannian Manifolds Without Curvature Constraints and Application to Riemannian Mass

Linear Convergence of Adaptive Stochastic Gradient Descent

Anisotropic Proximal Gradient

Linear Convergence of Primal-Dual Gradient Methods and their Performance in Distributed Optimization

Distributed Accelerated Proximal Coordinate Gradient Methods

An inexact variable metric variant of incremental aggregated Forward-Backward method for the large-scale composite optimization problem

Accelerated Primal-Dual Proximal Gradient Splitting Methods for Convex-Concave Saddle-Point Problems

Linear convergence of decentralized estimation for statistical estimation using gradient method

Accelerated Stochastic Algorithms for Nonconvex Finite-sum and Multi-block Optimization

Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Łojasiewicz Condition

On Convergence of Incremental Gradient for Non-Convex Smooth Functions

Linear convergence of accelerated conditional gradient algorithms in spaces of measures

On the Complexity Analysis of the Primal Solutions for the Accelerated Randomized Dual Coordinate Ascent

Global Convergence of a Grassmannian Gradient Descent Algorithm for Subspace Estimation

Last Iterate Convergence of Incremental Methods and Applications in Continual Learning

Iteration complexity analysis of random coordinate descent methods for $\ell_0$ regularized convex problems

Demystifying SGD with Doubly Stochastic Gradients