Abstract:We introduce a unified algorithmic framework, called the proximal-like incremental aggregated gradient (PLIAG) method, for minimizing the sum of a convex function that consists of additive relatively smooth convex components and a proper lower semicontinuous convex regularization function over an abstract feasible set whose geometry can be captured by using the domain of a Legendre function. The PLIAG method includes many existing algorithms in the literature as special cases, such as the proximal gradient method, the Bregman proximal gradient method (also called the NoLips algorithm), the incremental aggregated gradient method, the incremental aggregated proximal method, and the proximal incremental aggregated gradient method. It also includes some novel interesting iteration schemes. First, we show that the PLIAG method is globally sublinearly convergent without requiring a growth condition, which extends the sublinear convergence result for the proximal gradient algorithm to incremental aggregated-type first-order methods. Then, by embedding a so-called Bregman distance growth condition into a descent-type lemma to construct a special Lyapunov function, we show that the PLIAG method is globally linearly convergent in terms of both function values and Bregman distances to the optimal solution set, provided that the step size is not greater than some positive constant. The convergence results derived in this paper are all established beyond the standard assumptions in the literature (i.e., without requiring the strong convexity and the Lipschitz gradient continuity of the smooth part of the objective). When specialized to many existing algorithms, our results recover or supplement their convergence results under strictly weaker conditions.

Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Łojasiewicz Condition

Linear Convergence of the Proximal Gradient Method for Composite Optimization Under the Polyak-Łojasiewicz Inequality and Its Variant

Conditions for linear convergence of the gradient method for non-convex optimization

On the Linear Convergence of a Proximal Gradient Method for a Class of Nonsmooth Convex Minimization Problems

A Methodology Establishing Linear Convergence of Adaptive Gradient Methods under PL Inequality

Linear Convergence of Inexact Descent Method and Inexact Proximal Gradient Algorithms for Lower-Order Regularization Problems

On Convergence Rates of Linearized Proximal Algorithms for Convex Composite Optimization with Applications.

Linear Convergence Rate in Convex Setup is Possible! Gradient Descent Method Variants under $(L_0,L_1)$-Smoothness

Characterization of Linear Convergence of Gradient Descent

On Linear Convergence of Non-Euclidean Gradient Methods Without Strong Convexity and Lipschitz Gradient Continuity.

Linear Convergence of Generalized Mirror Descent with Time-Dependent Mirrors

Linear Convergence of Adaptive Stochastic Gradient Descent

A Linearly Convergent Conditional Gradient Algorithm with Applications to Online and Stochastic Optimization

Linear Convergence of Forward-Backward Accelerated Algorithms without Knowledge of the Modulus of Strong Convexity

Proximal Gradient Dynamics: Monotonicity, Exponential Convergence, and Applications

Linear convergence of accelerated conditional gradient algorithms in spaces of measures

Error bounds, PL condition, and quadratic growth for weakly convex functions, and linear convergences of proximal point methods

Proximal-Like Incremental Aggregated Gradient Method with Linear Convergence under Bregman Distance Growth Conditions

On the linear convergence rate of Riemannian proximal gradient method

The Global R-linear Convergence of Nesterov's Accelerated Gradient Method with Unknown Strongly Convex Parameter