Abstract: While the superior performance of second-order optimization methods such as Newton's method is well known, they are hardly used in practice for deep learning because neither assembling the Hessian matrix nor calculating its inverse is feasible for large-scale problems. Existing second-order methods resort to various diagonal or low-rank approximations of the Hessian, which often fail to capture necessary curvature information to generate a substantial improvement. On the other hand, when training becomes batch-based (i.e., stochastic), noisy second-order information easily contaminates the training procedure unless expensive safeguard is employed. In this paper, we adopt a numerical algorithm for second-order neural network training. We tackle the practical obstacle of Hessian calculation by using the complex-step finite difference (CSFD) -- a numerical procedure adding an imaginary perturbation to the function for derivative computation. CSFD is highly robust, efficient, and accurate (as accurate as the analytic result). This method allows us to literally apply any known second-order optimization methods for deep learning training. Based on it, we design an effective Newton Krylov procedure. The key mechanism is to terminate the stochastic Krylov iteration as soon as a disturbing direction is found so that unnecessary computation can be avoided. During the optimization, we monitor the approximation error in the Taylor expansion to adjust the step size. This strategy combines advantages of line search and trust region methods making our method preserves good local and global convergency at the same time. We have tested our methods in various deep learning tasks. The experiments show that our method outperforms exiting methods, and it often converges one-order faster. We believe our method will inspire a wide-range of new algorithms for deep learning and numerical optimization.

Adaptive Orthogonal Gradient Descent Algorithm for Fully Complex-Valued Neural Networks.

Is a Complex-Valued Stepsize Advantageous in Complex-Valued Gradient Learning Algorithms?

A Multilayer Complex Neural Network Training Algorithm and Its Application in Adaptive Equalization

Complex-valued Neurons Can Learn More but Slower Than Real-valued Neurons Via Gradient Descent.

Simple2Complex: Global Optimization by Gradient Descent

A Complex Hebbian-type Learning Algorithm with Application to Complex Adaptive ⅡR Filtering

A Split-Complex Valued Gradient-Based Descent Neuro-Fuzzy Algorithm for TS System and Its Convergence

Optimizing Double-Phase Method Based on Gradient Descent Algorithm with Complex Spectrum Loss Function

A L-BFGS Based Learning Algorithm for Complex-Valued Feedforward Neural Networks

An Input Weights Dependent Complex-Valued Learning Algorithm Based on Wirtinger Calculus

On The Unified Design Of Accelerated Gradient Descent

Adaptive Step Sizes for Preconditioned Stochastic Gradient Descent

Training Feed-forward Neural Networks Using the Gradient Descent Method with the Optimal Stepsize

Gradient Descent Optimization in Deep Learning Model Training Based on Multistage and Method Combination Strategy

Second-order Neural Network Training Using Complex-step Directional Derivative

ABNGrad: adaptive step size gradient descent for optimizing neural networks

Stochastic Primal–Dual Hybrid Gradient Algorithm with Adaptive Step Sizes

Novel Convergence Results of Adaptive Stochastic Gradient Descents

Stochastic Primal Dual Hybrid Gradient Algorithm with Adaptive Step-Sizes

Accelerated Gradient Approach For Deep Neural Network-Based Adaptive Control of Unknown Nonlinear Systems

Gradient Descent based Optimization Algorithms for Deep Learning Models Training