Abstract:Golden-section search and bisection search are the two main principled algorithms for 1d minimization of quasiconvex (unimodal) functions. The first one only uses function queries, while the second one also uses gradient queries. Other algorithms exist under much stronger assumptions, such as Newton's method. However, to the best of our knowledge, there is no principled exact line search algorithm for general convex functions -- including piecewise-linear and max-compositions of convex functions -- that takes advantage of convexity. We propose two such algorithms: $\Delta$-Bisection is a variant of bisection search that uses (sub)gradient information and convexity to speed up convergence, while $\Delta$-Secant is a variant of golden-section search and uses only function queries. While bisection search reduces the $x$ interval by a factor 2 at every iteration, $\Delta$-Bisection reduces the (sometimes much) smaller $x^*$-gap $\Delta^x$ (the $x$ coordinates of $\Delta$) by at least a factor 2 at every iteration. Similarly, $\Delta$-Secant also reduces the $x^*$-gap by at least a factor 2 every second function query. Moreover, the $y^*$-gap $\Delta^y$ (the $y$ coordinates of $\Delta$) also provides a refined stopping criterion, which can also be used with other algorithms. Experiments on a few convex functions confirm that our algorithms are always faster than their quasiconvex counterparts, often by more than a factor 2. We further design a quasi-exact line search algorithm based on $\Delta$-Secant. It can be used with gradient descent as a replacement for backtracking line search, for which some parameters can be finicky to tune -- and we provide examples to this effect, on strongly-convex and smooth functions. We provide convergence guarantees, and confirm the efficiency of quasi-exact line search on a few single- and multivariate convex functions.

A Line Search Multigrid Method for Large-Scale Nonlinear Optimization

A Line Search Multigrid Method for Large-Scale Convex Optimization

The convergence of conjugate gradient method with nonmonotone line search

An Efficient Interior-Point Algorithm with New Non-Monotone Line Search Filter Method for Nonlinear Constrained Programming

A Multilevel Low-Rank Newton Method with Super-linear Convergence Rate and its Application to Non-convex Problems

Modified Line Search Sequential Quadratic Methods for Equality-Constrained Optimization with Unified Global and Local Convergence Guarantees

The global convergence of the BFGS method with a modified WWP line search for nonconvex functions

An improvement of the Goldstein line search

A Conjugate Gradient Method with Global Convergence for Large-Scale Unconstrained Optimization Problems

An improved Dai‐Liao‐style hybrid conjugate gradient‐based method for solving unconstrained nonconvex optimization and extension to constrained nonlinear monotone equations

Line Search Filter Methods for Nonlinear Programming: Motivation and Global Convergence

An Efficient Algorithm for Large Scale Global Optimization of Continuous Functions

A New Hybrid Algorithm for Solving Large Scale Global Optimization Problems

Coderivative-Based Newton Methods with Wolfe Linesearch for Nonsmooth Optimization

Auto-conditioned primal-dual hybrid gradient method and alternating direction method of multipliers

An inertia-free filter line-search algorithm for large-scale nonlinear programming

A General Two-Level Subspace Method for Nonlinear Optimization

Non-asymptotic Global Convergence Rates of BFGS with Exact Line Search

A Modified Nonlinear Conjugate Gradient Algorithm for Large-Scale Nonsmooth Convex Optimization

Line Search for Convex Minimization

Line Search Fixed Point Algorithms Based on Nonlinear Conjugate Gradient Directions: Application to Constrained Smooth Convex Optimization