Abstract: While the superior performance of second-order optimization methods such as Newton's method is well known, they are hardly used in practice for deep learning because neither assembling the Hessian matrix nor calculating its inverse is feasible for large-scale problems. Existing second-order methods resort to various diagonal or low-rank approximations of the Hessian, which often fail to capture necessary curvature information to generate a substantial improvement. On the other hand, when training becomes batch-based (i.e., stochastic), noisy second-order information easily contaminates the training procedure unless expensive safeguard is employed. In this paper, we adopt a numerical algorithm for second-order neural network training. We tackle the practical obstacle of Hessian calculation by using the complex-step finite difference (CSFD) -- a numerical procedure adding an imaginary perturbation to the function for derivative computation. CSFD is highly robust, efficient, and accurate (as accurate as the analytic result). This method allows us to literally apply any known second-order optimization methods for deep learning training. Based on it, we design an effective Newton Krylov procedure. The key mechanism is to terminate the stochastic Krylov iteration as soon as a disturbing direction is found so that unnecessary computation can be avoided. During the optimization, we monitor the approximation error in the Taylor expansion to adjust the step size. This strategy combines advantages of line search and trust region methods making our method preserves good local and global convergency at the same time. We have tested our methods in various deep learning tasks. The experiments show that our method outperforms exiting methods, and it often converges one-order faster. We believe our method will inspire a wide-range of new algorithms for deep learning and numerical optimization.

A Second-Order Finite-Difference Method for Derivative-Free Optimization

A New Algorithm For Unconstrained Optimization Problem With The Form Of Sum Of Squares Minimization

New Methods for Parametric Optimization via Differential Equations

Derivative-Free Optimization via Finite Difference Approximation: An Experimental Study

Using second-order information in gradient sampling methods for nonsmooth optimization

Convergence ball of a new fourth-order method for finding a zero of the derivative

Improved Algorithms Based on Trust Region Framework for Solving Unconstrained Derivative Free Optimization Problems

Second-order Neural Network Training Using Complex-step Directional Derivative

Finding Second-Order Stationary Points in Nonconvex-Strongly-Concave Minimax Optimization

Distributed Optimization Algorithm with Superlinear Convergence Rate

Complexity-optimal and parameter-free first-order methods for finding stationary points of composite optimization problems

Newton-CG methods for nonconvex unconstrained optimization with Hölder continuous Hessian

Curvature-Aware Derivative-Free Optimization

Numerical Differentiation for the Second Order Derivatives of Functions of Two Variables

DRSOM: A Dimension Reduced Second-Order Method

Fast Convex Optimization via Differential Equation with Hessian-Driven Damping and Tikhonov Regularization

A successive difference-of-convex approximation method for a class of nonconvex nonsmooth optimization problems

Second-Order Optimization via Quiescence

$Q$-fully Quadratic Modeling and its Application in a Random Subspace Derivative-free Method

Q-fully quadratic modeling and its application in a random subspace derivative-free method

Second-Order Forward-Mode Automatic Differentiation for Optimization