Abstract:First-order algorithms have been popular for solving convex and non-convex optimization problems. A key assumption for the majority of these algorithms is that the gradient of the objective function is globally Lipschitz continuous, but many contemporary problems such as tensor decomposition fail to satisfy such an assumption. This paper develops a sequential local optimization (SLO) framework of first-order algorithms that can effectively optimize problems without Lipschitz gradient. Operating on the assumption that the gradients are {\it locally} Lipschitz continuous over any compact set, the proposed framework carefully restricts the distance between two successive iterates. We show that the proposed framework can easily adapt to existing first-order methods such as gradient descent (GD), normalized gradient descent (NGD), accelerated gradient descent (AGD), as well as GD with Armijo line search. Remarkably, the latter algorithm is totally parameter-free and do not even require the knowledge of local Lipschitz constants. We show that for the proposed algorithms to achieve gradient error bound of $\|\nabla f(x)\|^2\le \epsilon$, it requires at most $\mathcal{O}(\frac{1}{\epsilon}\times \mathcal{L}(Y))$ total access to the gradient oracle, where $\mathcal{L}(Y)$ characterizes how the local Lipschitz constants grow with the size of a given set $Y$. Moreover, we show that the variant of AGD improves the dependency on both $\epsilon$ and the growth function $\mathcal{L}(Y)$. The proposed algorithms complement the existing Bregman Proximal Gradient (BPG) algorithm, because they do not require the global information about problem structure to construct and solve Bregman proximal mappings.

A Local Nearly Linearly Convergent First-Order Method for Nonsmooth Functions with Quadratic Growth

A nearly linearly convergent first-order method for nonsmooth functions with quadratic growth

Gauss-Newton Methods for A Class of Nonsmooth Optimization Problems

Linear convergence of first order methods for non-strongly convex optimization

On the Local Convergence of a Stochastic Semismooth Newton Method for Nonsmooth Nonconvex Optimization

First Order Methods beyond Convexity and Lipschitz Gradient Continuity with Applications to Quadratic Inverse Problems

A descent subgradient method using Mifflin line search for nonsmooth nonconvex optimization

A linearly convergent Gauss-Newton subgradient method for ill-conditioned problems

A Stochastic Semismooth Newton Method for Nonsmooth Nonconvex Optimization.

A Modified Nonlinear Conjugate Gradient Algorithm for Large-Scale Nonsmooth Convex Optimization

First-Order Methods for Nonsmooth Nonconvex Functional Constrained Optimization with or without Slater Points

Non-Uniform Smoothness for Gradient Descent

Variational Analysis Perspective on Linear Convergence of Some First Order Methods for Nonsmooth Convex Optimization Problems

A descent subgradient method using Mifflin's line search for nonsmooth nonconvex optimization

First-Order Algorithms Without Lipschitz Gradient: A Sequential Local Optimization Approach

A New Inexact Gradient Descent Method with Applications to Nonsmooth Convex Optimization

Linear Convergence of Forward-Backward Accelerated Algorithms without Knowledge of the Modulus of Strong Convexity

Convex and Non-convex Optimization Under Generalized Smoothness

A globally convergent proximal Newton-type method in nonsmooth convex optimization

Faster Gradient-Free Algorithms for Nonsmooth Nonconvex Stochastic Optimization

A Proximal-Gradient Method for Constrained Optimization