Abstract:While Standard gradient descent is one very popular optimisation method, its convergence cannot be proven beyond the class of functions whose gradient is globally Lipschitz continuous. As such, it is not actually applicable to realistic applications such as Deep Neural Networks. In this paper, we prove that its backtracking variant behaves very nicely, in particular convergence can be shown for all Morse functions. The main theoretical result of this paper is as follows. Theorem. Let $f:\mathbb{R}^k\rightarrow \mathbb{R}$ be a $C^1$ function, and $\{z_n\}$ a sequence constructed from the Backtracking gradient descent algorithm. (1) Either $\lim _{n\rightarrow\infty}||z_n||=\infty$ or $\lim _{n\rightarrow\infty}||z_{n+1}-z_n||=0$. (2) Assume that $f$ has at most countably many critical points. Then either $\lim _{n\rightarrow\infty}||z_n||=\infty$ or $\{z_n\}$ converges to a critical point of $f$. (3) More generally, assume that all connected components of the set of critical points of $f$ are compact. Then either $\lim _{n\rightarrow\infty}||z_n||=\infty$ or $\{z_n\}$ is bounded. Moreover, in the latter case the set of cluster points of $\{z_n\}$ is connected. Some generalised versions of this result, including an inexact version, are included. Another result in this paper concerns the problem of saddle points. We then present a heuristic argument to explain why Standard gradient descent method works so well, and modifications of the backtracking versions of GD, MMT and NAG. Experiments with datasets CIFAR10 and CIFAR100 on various popular architectures verify the heuristic argument also for the mini-batch practice and show that our new algorithms, while automatically fine tuning learning rates, perform better than current state-of-the-art methods such as MMT, NAG, Adagrad, Adadelta, RMSProp, Adam and Adamax.

Backtracking New Q-Newton's method, Newton's flow, Voronoi's diagram and Stochastic root finding

Backtracking New Q-Newton's Method, Newton's Flow, Voronoi's Diagram and Stochastic Root Finding

Backtracking New Q-Newton's method, Schröder's theorem, and Linear Conjugacy

The Riemann hypothesis and dynamics of Backtracking New Q-Newton's method

Subspace Truncated-Newton Algorithm for Large-Scale Bound Constrained Optimization

Greedy Newton: Newton's Method with Exact Line Search

A Stochastic Extra-Step Quasi-Newton Method for Nonsmooth Nonconvex Optimization

A New Backtracking Inexact Bfgs Method for Symmetric Nonlinear Equations

Adaptive Greedy Quasi-Newton with Superlinear Rate and Global Convergence Guarantee

A Two-Point Newton Method suitable for non-convergent Cases and with Super-Quadratic Convergence

A novel Newton-Raphson style root finding algorithm

Stochastic Quasi-Newton Optimization in Large Dimensions Including Deep Network Training

Backtracking gradient descent method for general $C^1$ functions, with applications to Deep Learning

A randomized Newton's method for solving differential equations based on the neural network discretization

Revisiting Sub-sampled Newton Methods

Asynchronous Parallel Stochastic Quasi-Newton Methods

Faster Explicit Superlinear Convergence for Greedy and Random Quasi-Newton Methods

Quasi-Newton Methods for Topology Optimization Using a Level-Set Method

Distributed Adaptive Greedy Quasi-Newton Methods with Explicit Non-asymptotic Convergence Bounds

Root-finding: from Newton to Halley and beyond

New high-order convergence iteration methods without employing derivatives for solving nonlinear equations