Abstract: While the superior performance of second-order optimization methods such as Newton's method is well known, they are hardly used in practice for deep learning because neither assembling the Hessian matrix nor calculating its inverse is feasible for large-scale problems. Existing second-order methods resort to various diagonal or low-rank approximations of the Hessian, which often fail to capture necessary curvature information to generate a substantial improvement. On the other hand, when training becomes batch-based (i.e., stochastic), noisy second-order information easily contaminates the training procedure unless expensive safeguard is employed. In this paper, we adopt a numerical algorithm for second-order neural network training. We tackle the practical obstacle of Hessian calculation by using the complex-step finite difference (CSFD) -- a numerical procedure adding an imaginary perturbation to the function for derivative computation. CSFD is highly robust, efficient, and accurate (as accurate as the analytic result). This method allows us to literally apply any known second-order optimization methods for deep learning training. Based on it, we design an effective Newton Krylov procedure. The key mechanism is to terminate the stochastic Krylov iteration as soon as a disturbing direction is found so that unnecessary computation can be avoided. During the optimization, we monitor the approximation error in the Taylor expansion to adjust the step size. This strategy combines advantages of line search and trust region methods making our method preserves good local and global convergency at the same time. We have tested our methods in various deep learning tasks. The experiments show that our method outperforms exiting methods, and it often converges one-order faster. We believe our method will inspire a wide-range of new algorithms for deep learning and numerical optimization.

HoD-Net: High-Order Differentiable Deep Neural Networks and Applications.

SHoP: A Deep Learning Framework for Solving High-order Partial Differential Equations

Convolution Hierarchical Deep-learning Neural Networks (C-HiDeNN): finite elements, isogeometric analysis, tensor decomposition, and beyond

Hierarchical Deep Learning Neural Network (HiDeNN): An artificial intelligence (AI) framework for computational science and engineering

Hierarchical Deep-Learning Neural Networks: Finite Elements and Beyond

HiDeNN-TD: Reduced-order Hierarchical Deep Learning Neural Networks

Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations

DOF: Accelerating High-order Differential Operators with Forward Propagation

Convolutional Hierarchical Deep Learning Neural Networks-Tensor Decomposition (C-HiDeNN-TD): a scalable surrogate modeling approach for large-scale physical systems

Second-order Neural Network Training Using Complex-step Directional Derivative

HNO: High-Order Numerical Architecture for ODE-Inspired Deep Unfolding Networks.

Derivative-enhanced Deep Operator Network

Differentiable homotopy methods for gradually reinforcing the training of fully connected neural networks

A Deep Neural Network Framework for Solving Forward and Inverse Problems in Delay Differential Equations

Convolution hierarchical deep-learning neural network (C-HiDeNN) with graphics processing unit (GPU) acceleration

Enhancing Convolutional Neural Networks with Higher-Order Numerical Difference Methods

Toward Deep Adaptive Hinging Hyperplanes

The advance of neural ordinary differential ordinary differential equations

A High Resolution Physics-informed Neural Networks for High-Dimensional Convection-Diffusion-reaction Equations.

A Non-Gradient Method for Solving Elliptic Partial Differential Equations with Deep Neural Networks.

HiDeS: a higher-order-derivative-supervised neural ordinary differential equation for multi-robot systems and opinion dynamics