Abstract: While the superior performance of second-order optimization methods such as Newton's method is well known, they are hardly used in practice for deep learning because neither assembling the Hessian matrix nor calculating its inverse is feasible for large-scale problems. Existing second-order methods resort to various diagonal or low-rank approximations of the Hessian, which often fail to capture necessary curvature information to generate a substantial improvement. On the other hand, when training becomes batch-based (i.e., stochastic), noisy second-order information easily contaminates the training procedure unless expensive safeguard is employed. In this paper, we adopt a numerical algorithm for second-order neural network training. We tackle the practical obstacle of Hessian calculation by using the complex-step finite difference (CSFD) -- a numerical procedure adding an imaginary perturbation to the function for derivative computation. CSFD is highly robust, efficient, and accurate (as accurate as the analytic result). This method allows us to literally apply any known second-order optimization methods for deep learning training. Based on it, we design an effective Newton Krylov procedure. The key mechanism is to terminate the stochastic Krylov iteration as soon as a disturbing direction is found so that unnecessary computation can be avoided. During the optimization, we monitor the approximation error in the Taylor expansion to adjust the step size. This strategy combines advantages of line search and trust region methods making our method preserves good local and global convergency at the same time. We have tested our methods in various deep learning tasks. The experiments show that our method outperforms exiting methods, and it often converges one-order faster. We believe our method will inspire a wide-range of new algorithms for deep learning and numerical optimization.

ISAAC Newton: Input-based Approximate Curvature for Newton's Method

Newton Losses: Using Curvature Information for Learning with Differentiable Algorithms

Optimizing Neural Networks with Kronecker-factored Approximate Curvature

EA-CG: An Approximate Second-Order Method for Training Fully-Connected Neural Networks

Kronecker-Factored Approximate Curvature for Physics-Informed Neural Networks

Gradient Descent on Neurons and its Link to Approximate Second-Order Optimization

Ginger: An Efficient Curvature Approximation with Linear Complexity for General Neural Networks

Inertial Newton Algorithms Avoiding Strict Saddle Points

Series of Hessian-Vector Products for Tractable Saddle-Free Newton Optimisation of Neural Networks

Second-order Neural Network Training Using Complex-step Directional Derivative

Stochastic Sub-Sampled Newton Method with Variance Reduction

A Trace-restricted Kronecker-Factored Approximation to Natural Gradient

Adapting Newton's Method to Neural Networks through a Summary of Higher-Order Derivatives

Practical Quasi-Newton Methods for Training Deep Neural Networks

M-FAC: Efficient Matrix-Free Approximations of Second-Order Information

On Newton's Method to Unlearn Neural Networks

A Computationally Efficient Sparsified Online Newton Method

Quadratic Gradient: Combining Gradient Algorithms and Newton's Method as One

Preconditioners for the Stochastic Training of Implicit Neural Representations

Nesterov's Acceleration For Approximate Newton.

Stochastic Second-order Methods for Non-convex Optimization with Inexact Hessian and Gradient