Abstract:In this paper, we consider a class of finite-sum convex optimization problems whose objective function is given by the average of m(≥1)\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$m\, ({\ge }1)$$\end{document} smooth components together with some other relatively simple terms. We first introduce a deterministic primal–dual gradient (PDG) method that can achieve the optimal black-box iteration complexity for solving these composite optimization problems using a primal–dual termination criterion. Our major contribution is to develop a randomized primal–dual gradient (RPDG) method, which needs to compute the gradient of only one randomly selected smooth component at each iteration, but can possibly achieve better complexity than PDG in terms of the total number of gradient evaluations. More specifically, we show that the total number of gradient evaluations performed by RPDG can be O(m)\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$${{\mathcal {O}}} (\sqrt{m})$$\end{document} times smaller, both in expectation and with high probability, than those performed by deterministic optimal first-order methods under favorable situations. We also show that the complexity of the RPDG method is not improvable by developing a new lower complexity bound for a general class of randomized methods for solving large-scale finite-sum convex optimization problems.

Relaxed Proximal Point Algorithm: Tight Complexity Bounds and Acceleration without Momentum

A UNIFIED AND TIGHT LINEAR CONVERGENCE ANALYSIS OF THE RELAXED PROXIMAL POINT ALGORITHM

An Accelerated Distributed Method with Inexact Model of Relative Smoothness and Strong Convexity

A Symplectic Discretization Based Proximal Point Algorithm for Convex Minimization

An adaptive proximal point algorithm framework and application to large-scale optimization

On the optimal linear convergence factor of the relaxed proximal point algorithm for monotone inclusion problems.

Accelerating Proximal Gradient Descent via Silver Stepsizes

Accelerated Quasi-Newton Proximal Extragradient: Faster Rate for Smooth Convex Optimization

General Parameterized Proximal Point Algorithm with Applications in Statistical Learning

Asynchronous Stochastic Proximal Methods for Nonconvex Nonsmooth Optimization.

An Indefinite-Proximal-based Strictly Contractive Peaceman-Rachford Splitting Method

A Simple Proximal Stochastic Gradient Method for Nonsmooth Nonconvex Optimization.

Accelerated Bregman Proximal Gradient Methods for Relatively Smooth Convex Optimization

New nonasymptotic convergence rates of stochastic proximal pointalgorithm for convex optimization problems

Accelerated Stochastic Algorithms for Nonconvex Finite-sum and Multi-block Optimization

Distributed Proximal Point Algorithm For Constrained Optimization Over Unbalanced Graphs

A Bregman Proximal Stochastic Gradient Method with Extrapolation for Nonconvex Nonsmooth Problems

A Proximal-Gradient Method for Constrained Optimization

Proximal point type algorithms with relaxed and inertial effects beyond convexity

An optimal randomized incremental gradient method

Complexity of Inexact Proximal Point Algorithm for minimizing convex functions with Holderian Growth