Abstract:Golden-section search and bisection search are the two main principled algorithms for 1d minimization of quasiconvex (unimodal) functions. The first one only uses function queries, while the second one also uses gradient queries. Other algorithms exist under much stronger assumptions, such as Newton's method. However, to the best of our knowledge, there is no principled exact line search algorithm for general convex functions -- including piecewise-linear and max-compositions of convex functions -- that takes advantage of convexity. We propose two such algorithms: $\Delta$-Bisection is a variant of bisection search that uses (sub)gradient information and convexity to speed up convergence, while $\Delta$-Secant is a variant of golden-section search and uses only function queries. While bisection search reduces the $x$ interval by a factor 2 at every iteration, $\Delta$-Bisection reduces the (sometimes much) smaller $x^*$-gap $\Delta^x$ (the $x$ coordinates of $\Delta$) by at least a factor 2 at every iteration. Similarly, $\Delta$-Secant also reduces the $x^*$-gap by at least a factor 2 every second function query. Moreover, the $y^*$-gap $\Delta^y$ (the $y$ coordinates of $\Delta$) also provides a refined stopping criterion, which can also be used with other algorithms. Experiments on a few convex functions confirm that our algorithms are always faster than their quasiconvex counterparts, often by more than a factor 2. We further design a quasi-exact line search algorithm based on $\Delta$-Secant. It can be used with gradient descent as a replacement for backtracking line search, for which some parameters can be finicky to tune -- and we provide examples to this effect, on strongly-convex and smooth functions. We provide convergence guarantees, and confirm the efficiency of quasi-exact line search on a few single- and multivariate convex functions.

A More Lenient Stopping Rule for Line Search Algorithms

The convergence of conjugate gradient method with nonmonotone line search

An Efficient Interior-Point Algorithm with New Non-Monotone Line Search Filter Method for Nonlinear Constrained Programming

Line Search for Convex Minimization

Variable metric inexact line-search based methods for nonsmooth optimization

A new stopping criterion for Krylov solvers applied in Interior Point Methods

Universal nonmonotone line search method for nonconvex multiobjective optimization problems with convex constraints

Convergence Conditions for Stochastic Line Search Based Optimization of Over-parametrized Models

A descent subgradient method using Mifflin's line search for nonsmooth nonconvex optimization

Convergence depth control for interior point methods

A class of nonmonotone Armijo-type line search method for unconstrained optimization

Modified Armijo line-search in Riemannian optimization with reduced computational cost

Efficient Line Search Method Based on Regression and Uncertainty Quantification

A nonmonotone line search method for noisy minimization

A descent subgradient method using Mifflin line search for nonsmooth nonconvex optimization

A new proximal heavy ball inexact line-search algorithm

A Subsampling Line-Search Method with Second-Order Results

A Parallel Line Search Subspace Correction Method for Composite Convex Optimization

Don't be so Monotone: Relaxing Stochastic Line Search in Over-Parameterized Models

Stopping criteria for nonlinear variational problems: iterative approach

A Line Search Multigrid Method for Large-Scale Nonlinear Optimization